

IE10: How does it really stack up? - pughbri
http://www.lucidchart.com/blog/2012/11/14/ie10-how-does-it-really-stack-up/

======
SideburnsOfDoom
The real problem is that while there will be a new and better Chrome and in a
month and a half, and similar for Firefox, we'll have this build of IE10 for 2
years or maybe more.

How they compare at the start of the cycle is not typical, how they compare at
the midpoint will be more average, how they compare at the end (where IE7 or
IE8 is now) is the worst case.

~~~
Jabbles
Internet Explorer will start automatically updating.

[http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-
st...](http://blogs.windows.com/ie/b/ie/archive/2011/12/15/ie-to-start-
automatic-upgrades-across-windows-xp-windows-vista-and-windows-7.aspx)

~~~
polshaw
The question is how often. Adopting the firefox 2/3 model (albeit less nagging
a-la-chrome) is a great step forward from the IE6 one we have had up to now,
but they really need to match the rapid release schedules of mozilla and
google today. At a minimum quarterly updates (features/engine updates, not
just security fixes) to stop hindering the web.

~~~
vegardx
Less naggy? I find Firefox to be one of the most naggy browsers about
upgrades. Or have they changed this recently? With Chrome you just see a
notification on the settings menu, and a restart brings you up to the next
version. I really hope IE adopts this model.

~~~
thristian
I'm pretty sure recent versions of Firefox (say, since the early teens) have
had nagless updates.

I find myself having to manually check the About dialog for updates now,
because they're about the only reason I ever restart my browser, and if I
don't manually check occasionally I'd have no way of knowing an updated
existed.

------
ck2
_We gathered timing information for the parts of the benchmark that it did
complete_

Which parts didn't complete? Did you try the beta or aurora?

Code that can consistently crash firefox would probably be elgible for a
security bounty.

~~~
bendilts
Interestingly, the part that crashed Firefox was one of the simpler tests--
typing a bunch of text content in, then resizing and rotating the shapes that
contain that text. Firefox didn't give any details as to why it crashed.

The only thing I could even measure that would lead to a crash is memory
usage. I just re-ran the test and watched the process's memory usage. It
hovered in the 300-400MB range during the first third of the test. Then,
during the test I describe, memory usage rocketed to around 1GB, at which
point the browser crashed hard.

I'm not sure why Firefox would exhibit this behavior when other browsers
don't, but we know a few people at Mozilla (they're customers of ours) so
we'll probably reach out.

~~~
carlosreyes
Why not remove the part of the test that crashed it and post results of the
rest of the benchmarks? Those are 3 tests out of 16, I would still like to see
how it compares in the other benchmark tests.

Seeing "We're sorry. Firefox had a problem and crashed." as well as its
exclusion from the performance summary made it feel like there was a bit of a
bias against Firefox, where even in your conclusion it's stated that it
performs "quite well" - that sentence does not reflect on the rest of the
article at all.

------
monsterix
Well, let's agree that IE10 does well whatever it is designed to do well.
Cool. Now what? We have an IE10 browser that compares well with Google Chrome,
Opera and perhaps Firefox too (I don't know the exact details about your tests
that were making Firefox crash as often as you've claimed in your article).

So is this the point where Microsoft takes on the real challenge? Will it go
beyond its typical high-speed sales pitch for the new browser and attack the
more relevant problem that is to really get those IE6/IE7/IE8 and perhaps even
IE9 users to adopt IE10 immediately?

Just a few days ago, I saw on HN how IE6 has become a political election issue
in South Korea!

Pushing the lazy and entrenched users out of the hole is a much more important
(and harder) problem to solve than to win medals over browser speed and
performance tests. Even if tests like these help push the cause of IE10, it
would be appropriate to give out all details of the tests and realize why
Firefox was 'disagreeing' to your test at all?

~~~
bornhuetter
How do you suggest Microsoft goes about convincing the millions of East Asian
users running IE6 on a pirated version of XP to upgrade to IE10?

~~~
WiseWeasel
Release IE10 for WinXP, and don't check authentication.

~~~
ygra
Given that IE 10 (and 9 before it) builds on core Windows APIs only introduced
in Windows 7 I'd consider that unlikely. And they definitely won't invest the
time and effort of backporting DirectWrite, Direct2D and whatnot to XP (this
might even include DirectX 10 and/or 11 along with WARP) which is already out
of mainstream support and only get security patches.

------
simonster
This should really be titled "IE10: How does it really stack up in one
benchmark."

~~~
pbhjpbhj
Not just one benchmark, but one apparently completely undocumented benchmark
run under Windows 8 on a single laptop.

They don't mention how many times the benchmark ran which makes me suspicious
that the answer might be one??

Unless they release the benchmarking tool then this result seems pretty
worthless to me.

> _we’ve developed a benchmark that runs inside of the Lucidchart diagramming
> application_ //

So maybe that line for Firefox could as well read "Lucidchart crashed"?

~~~
untog
_So maybe that line for Firefox could as well read "Lucidchart crashed"?_

No, I'd say that JS should never crash the browser, no matter what.

~~~
pbhjpbhj
OK fair enough but if I code something wrongly it doesn't work and fails badly
then I'm at least partly to be blame, no?

It's moot anyway as it appears the test is proprietary and so can't be
investigated further?

------
VengefulCynic
_There is one more strange thing about IE10 that I should mention. IE9 often
maxes out one CPU core when a Lucidchart diagram is open, even when the user
is completely idle. IE10 is somewhat better, but on our test machine it still
used about 50% of one core when the user is idle. No other browser has any
detectable CPU load under the same conditions._

Has anyone else seen behavior similar to this in their own javascripting (is
that even a word?)? I hadn't been paying special attention to IE's CPU usage
but I'd like to think I would have noticed this in any of my own stuff when I
was doing browser compatibility testing.

~~~
digitalinfinity
According to [http://blogs.msdn.com/b/ie/archive/2012/06/13/advances-in-
ja...](http://blogs.msdn.com/b/ie/archive/2012/06/13/advances-in-javascript-
performance-in-ie10-and-windows-8.aspx), IE has separate GC and JIT threads-
if I were to guess, I bet it's the GC

------
striderxiii
IE10 had to skip a bunch of frames and use half a core to not quite keep up
with chrome and roughly match Firefox... Well, I suppose its an improvement
still, but they had a pretty low bar to begin with.

~~~
0xABADC0DA
In IE10 the JavaScript is multithreaded... there's a separate thread for JS
parsing, compiling, and garbage collection. As far as I know this is the only
browser with a threaded JavaScript VM. So the skipping frames is probably
because the rendering can be done independently from running JavaScript, and
the core in use is probably some quirk like the benchmark creating a few
objects on a timer, causing the GC to run on another thread -- that kind of
thing can be fixed pretty easily, either by tweaking the benchmark (if it were
publicly available) or by Microsoft fixing IE.

I haven't actually used IE10 yet, but I think you'll find this new
architecture makes overall browsing more responsive and smoother than Chrome.

------
olliesaunders
Opera used to market itself almost exclusively on how it was faster than any
other comparable browser. I just had a look on the Opera website and now it
says "Faster & Safer internet" yet it performed worst on this test. As the
underdog in the browser market, despite being what has always seemed to me as
a pretty nice piece of software, I feel a bit sorry for them. I hope they can
do whatever it takes to catch up with Chrome on the performance front.

~~~
pbhjpbhj
> _a look on the Opera website and now it says "Faster & Safer internet" yet
> it performed worst on this test._ //

Which page were you on, the front page of the website doesn't say that to me?

My first instinct would be that they're referring to their "turbo"
implementation and their large mobile install base, cf
<http://www.opera.com/browser/turbo/>.

~~~
olliesaunders
> Which page were you on, the front page of the website doesn't say that to
> me?

It's the contents of the <title> tag so it displays on the title bar of the
browser you are using to read it with and in the tab if you are using a
browser with tabs.

~~~
pbhjpbhj
Lol, I don't normally have a title bar (turned off for main FF window) and the
relevant part was most likely hidden by having a dozen or so tags open. Thanks
answering.

------
emehrkay
Were all of these tests done on a Windows8 machine? I only ask because the
latest version of Safari isnt available for Windows (you may be able to
install WebKit nightly as a Windows user though).

Id like to see how nightly build of browsers stack up as well.

~~~
bendilts
That's right, these were all done on a Windows 8 machine. We could of course
test on Safari 6 on a Mac, but then the numbers wouldn't be directly
comparable (since it's different hardware).

~~~
olliesaunders
I think your article would benefit from making this piont clearer. There's
been discussions on HN before about how performance of Apple software on
Windows differs greatly compared with performance of Apple software on Apple
OSes.

------
btipling
Others have already mentioned the probably slow pace of IE updates. Another
thing I'll mention is that testing IE requires Windows, and I haven't worked
at a place in years where anyone on my team was using Windows. Years. I've
worked at Ask.com, Cloudkick, Rackspace, and now a new startup. Until I can
test IE on a Mac, I wont bother to support it unless customers mention
something is broken, and they rarely do. Because few use IE.

~~~
ygra
You may have heard of this new thing called “Virtual Machine”.

~~~
btipling
My development environment runs on a VM. You can't run multiple VMs on a host.
I actually didn't want to reply to you because of your sarcasm. Also your
comment history is very windows focused.

~~~
ygra
The only time I ever had problems running multiple VMs at once was when I used
two different VMMs to do so. In fact, testing IE on Windows is equally painful
as you need the VMs for the older versions anyway (unless you happen to have a
few real machines lying around).

Depending on your site's demographics you may choose to ignore IE testing, I
understand. But often that demographic is rather disconnected from your own
development OS so you may not always get around it.

Also your comment history is very JS-focused (but how does that matter in any
way?).

~~~
btipling
My comment history is more varied I think. My development environment runs a
lot of stuff, so even if running two vm's at once were realistic, it wouldn't
be.

------
bhanks
My skills are a little rough and this is probably more than a company can
devote to a browser war study but it would be interesting to see this
benchmark run multiple times or as a Monte Carlo simulation.

Also - does anyone know what core runs at 50% in IE even when the user is
idle? Not surprised to hear it but curious if anyone knows what it is.

~~~
ygra
While idle, the profiler tells me that the function with highest time is yp at
line 5198 in chart.js. But no idea whether that's really the culprit.

~~~
bendilts
Hi, Lucid CTO here. We've spent a lot of time in the IE profiler, and it never
appears that any of our Javascript is consuming much CPU at all. There are a
handful of functions, like the one you quoted, that poll frequently on a
setInterval. But none of them are doing much more than checking a variable and
returning almost every time. It's really bizarre.

------
zobzu
The benchmark is "internal only". How convenient. So basically those are just
meaningless pretty graphics.

------
Yaggo
Benchmarking Safari under Windows pretty meaningless.

------
paranoiacblack
Wait, did you make the other browsers your default before running the tests? I
was under the impression that IE is the only first-class browser and the other
browsers would work as third-party apps with limitations on things like the
amount of processes it can open, etc. I ask because I supremely doubt that
Mozilla Firefox couldn't pass your benchmark, although I don't use it myself.

~~~
ygra
There are no limitations on desktop applications in Windows 8 whatsoever.

