
Performance Comparison of JavaScript Frameworks - etimberg
http://www.stefankrause.net/js-frameworks-benchmark2/webdriver-java/table.html
======
fenomas
Every time I see JS performance data like this, I'm always mystified to see
lots of error ranges of a couple percent. Working on a JS game has made me
dive pretty deeply into JS performance (on Chrome at least), and I constantly
find that the hardest part of performance tuning is getting any kind of
consistent results to tune.

For a random example, I tried reproducing the "Swap rows" test for Vanilla.js.
The author found this took 7.83 +/\- 0.18 ms, with five warmup iterations. On
my machine, running the same command six times consistently gives results like
these:

    
    
        swapRows took 10.944999999999709
        swapRows took 1.3250000000007276
        swapRows took 1.3200000000006185
        swapRows took 1.269999999999527
        swapRows took 11.875
        swapRows took 1.2750000000005457
    

Also, and orthogonal to the above, I've always found that even when
performance is consistent across repeated trials, it varies widely between
page reloads. (This isn't at all surprising, since v8's optimizing compiler
isn't necessarily going to make the same choices every time - but it still
makes things very hard to tune).

As an example of this, I tried repeating the "append to large table" test. The
author's results were 225.3 +/\- 1.89ms. On my (similar) machine, trying this
five times (reloading the page between) varied between 225 and 329ms.

Am I missing something here? I've looked pretty hard into v8 perf tuning and
various benchmarking libraries, and I've just never seen the kind of
consistent results one always finds in comparisons like these.

~~~
krausest
You're basically right. Thus we've decided to perform 10 runs for each
benchmark and drop the worst 4. This eliminates cases where the gc or a
background system process causes a slow run. In consequence this also strongly
reduces variance, but is blind on the gc eye. It would be interesting to
include gc, but I have no idea how to do it in a real good way.

The results are comparable (but nowhere equal) between runs and the difference
between the frameworks is in most cases large enough that the ranking stays
consistent (but you wouldn't prefer preact to cycle.js v7 for having a 0.01
better result in the average slow down, would you? For ember, mithril and
cycle.js v6 performance might be something to consider depending on your use
case.).

If you look at the console output this is just an approximation, the real
measurement is performed on data from chrome's timeline using selenium. The
console measurement uses a timer to guess when rendering was performed. Of
course this is more inaccurate than using the timeline.

~~~
yabb
Do the benchmarks take a long time? Why not as many runs as necessary to give
you a tight confidence interval?

~~~
krausest
Running all the benchmarks takes a few hours.

I'll keep that in mind for the next round. Let's see if that works nicely.
Would you still drop outliers?

------
seanwilson
Link to the blog post explaining the table:
[http://www.stefankrause.net/wp/?p=301](http://www.stefankrause.net/wp/?p=301)

I've worked on a few apps that feature large tables like this benchmark does.
With Angular for example, it's a really bad idea for performance to create DOM
elements for each table row when most aren't even visible which I think is
done in the benchmark: [https://github.com/krausest/js-framework-
benchmark/blob/mast...](https://github.com/krausest/js-framework-
benchmark/blob/master/angular-v1.5.7/index.html)

There's lots of grid/table libraries available that use what's called
virtualisation to stop lots of rows slowing the app down; rows that aren't on
the screen aren't computed or rendered.

My feeling is if you avoid rendering data excessively in this way the
performance differences between most frameworks aren't that important. If
you're required to display 1000s of dynamic values at a time your UI probably
needs work.

------
rattray
I found this chart unusable without being able to hide columns, so I created
this quick script to enable toggling by clicking the headers:

[https://gist.github.com/rattrayalex/8450e2378465e85ed846837a...](https://gist.github.com/rattrayalex/8450e2378465e85ed846837a5d0114a6)

copy/paste into chrome console

------
00098345
Just for reference, the fastest framework can be found here:
[Vannilajs]([http://vanilla-js.com/](http://vanilla-js.com/)) .

~~~
EdSharkey
VanillaJs is notable for how long it has been maintained, like 20+ years now!

I wish the implementers would go ahead and do a full rewrite, but there are
probably too many entrenched users who would have a fit if they did...

------
venning
This might be a better link, which includes a brief write-up of the work:
[http://www.stefankrause.net/wp/?p=301](http://www.stefankrause.net/wp/?p=301)

~~~
etimberg
Agreed. Hopefully one of the mods can change it

------
SwellJoe
I usually come away from things like this with the distinct impression that
they're pretty much all "fast enough" for my needs. Admittedly, I'm not asking
a lot of JavaScript, and my skill level with JavaScript tends to lead me to
take the path of least resistance, or at least the path of best documentation
and most examples.

That said, this kind of third-party benchmarking is probably useful, because
so many frameworks have benchmark charts on their own site which indicate
they're all faster than everything else. (Which may not even be
inaccurate...for the things they benchmark, they may be the fastest. Which
again maybe indicates they're all "fast enough" and unless your use case
exactly matches that of the specific benchmarks in question, the benchmarks
might be misleading you about which one will be fastest for your app.)

Also, I had no idea there were so many JavaScript frameworks that I hadn't
heard of (and there's at least a few that I have heard of that aren't on the
chart). Every time I turn around, there's two or three more JavaScript
frameworks. Surely, this will calm down at some point, and a couple of leaders
will emerge and folks will kinda settle into moving those leaders forward
rather than reinventing everything?

~~~
shados
It's already starting to calm down. On that list, you have a few broad
categories, with some overlap.

Angular and Angular-ish (I use the term loosely) frameworks like Aurelia.

Observable based frameworks (Cyclejs, Angular 2?)

React and React near-dropin replacements (Preact, React, Inferno, etc)

A bunch of MVC/MVVM frameworks that are not Angular-like (Mithril, Vue,
Ember...). This is where there's the most variety, namely because many of
these are a bit older (even though many kept up with time).

So there's not really nearly as many as you'd think, and in some cases
migrating from one to the other as need be isn't crazy.

~~~
kabes
I find Vue being very angular-like.

~~~
st553
Same here

------
bikamonki
_DOM manipulation_ should be added to the title and before calling one faster
than other one should understand that a JS framework is (should be) a lot more
than painting views super fast.

------
nnx
Any chance/interest at adding [http://riotjs.com](http://riotjs.com) to this
benchmark?

~~~
krausest
I added riot 2.5, but I need help. Can you assist?
[https://github.com/krausest/js-framework-
benchmark/issues/13](https://github.com/krausest/js-framework-
benchmark/issues/13)

------
collyw
I am not trying to be snarky with this question, but does it really matter
with JS? Its always going to be run in a browser by one user, not having
multiple people connecting to it like the server side does. As long as it
seems snappy to the user it shouldn't make a great deal of difference. Counter
opinions welcome.

------
beefsack
I feel like these comparisons are interesting but a bit one dimensional. A
broad generalisation you could make is that frameworks with more features tend
to be slower, but mean developers have to write less code.

A LOC or character count for each benchmark would help add a bit of context.

~~~
ben_jones
I agree with some of that, but I'd be more interested not in how much code is
written for the test, but how far into the private API the code must go before
the actual action is done. Kind of like using Test Coverage techniques. I
imagine theirs only like 2 or 3 ways developers are, say, appending lots of
rows, and the difference maker is how efficiently those techniques are being
implemented.

------
SimeVidas
This is benchmark porn. You can create a fast-performing web app with _any_ of
these frameworks, as long as you focus on the right metrics, like “time to
interact”.

------
myrryr
I'm really surprised that d3.js isn't on the list. It should be. I _suspect_
it would completely own most of the tests, but be bad on memory allocation.

------
vassy
Ember looks really bad.

~~~
shados
I'm not a fan of Ember, but it's focus on tooling, developer experience, and
sticking to it's core principles in the name of consistency is something to be
admired. It's not as shiny as Cycle or React, but if you started working with
Ember years ago, you've had reasonably robust tooling all along (as opposed to
the bleeding edge constantly breaking crap of the other frameworks), and have
been able to reuse your knowledge for years, so you're potentially quite
productive.

At least it's the impression I get.

~~~
lossolo
There was a lot of breaking the world changes until 2.1. I am using Ember from
2014. It stabilized lately so no breaking changes.

------
zerr
I've heard good things about the performance of Knockout.js - isn't this
popular anymore?

~~~
ak39
Knockout not considered "framework", just a lowly "library". Like jquery.

Frameworks are those that force their own opinion/style on your code structure
and architecture.

~~~
paublyrne
But isn't React also a library rather than a framework by this measure? I
think knockout is no longer in development so like Backbone it doesn't really
merit inclusion.

~~~
ww520
That's news to me. The last release was just couple months ago.

Knockout is pretty much feature stable with most stuff in place, so it won't
have as frequent releases as younger frameworks.

------
rajanchandi
I'd have liked to see modern things like nunjucks and marko in this comparison
list.

~~~
porker
> modern things

Heh, so a bunch of unreleased beta-frameworks don't count as modern any more?

------
api
These frameworks are all so different it renders these results pretty moot.
Vanillaja is fastest but it doesn't offer half of what React or Angular do.

~~~
niutech
Actually everything you do in Angular or React you can do in vanilla JS, so
your point is moot.

~~~
api
Everything you can do in Python you can do in assembly language.

------
ww520
How is Knockout.js compared to these?

~~~
diziet
I welcome you to add a pr for knockout.

Take a look at the sample implementations here:

[https://github.com/krausest/js-framework-
benchmark/blob/mast...](https://github.com/krausest/js-framework-
benchmark/blob/master/vanillajs/src/Main.js)

[https://github.com/krausest/js-framework-
benchmark/blob/mast...](https://github.com/krausest/js-framework-
benchmark/blob/master/angular-v1.5.7/src/HomeController.es6.js)

~~~
pbowyer
I've just finished my first project in KnockoutJS, I'm up for contributing to
a Knockout version.

I'd like to collaborate with others, so we don't end up with 3 different
Knockout implementations ;-)

~~~
diziet
Ok, let me regex your email out and I'll ping you.

------
pier25
Why is React performance slowing?

~~~
davej
I suspect there's a perf bug in React 15 that causes the 'clear rows' test to
be an order of magnitude slower than react 0.14. Otherwise, React is getting
faster.

edit: My hunch was correct, see here
[https://github.com/facebook/react/issues/7227](https://github.com/facebook/react/issues/7227)

