For a random example, I tried reproducing the "Swap rows" test for Vanilla.js. The author found this took 7.83 +/- 0.18
ms, with five warmup iterations. On my machine, running the same command six times consistently gives results like these:
swapRows took 10.944999999999709
swapRows took 1.3250000000007276
swapRows took 1.3200000000006185
swapRows took 1.269999999999527
swapRows took 11.875
swapRows took 1.2750000000005457
As an example of this, I tried repeating the "append to large table" test. The author's results were 225.3 +/- 1.89ms. On my (similar) machine, trying this five times (reloading the page between) varied between 225 and 329ms.
Am I missing something here? I've looked pretty hard into v8 perf tuning and various benchmarking libraries, and I've just never seen the kind of consistent results one always finds in comparisons like these.
The results are comparable (but nowhere equal) between runs and the difference between the frameworks is in most cases large enough that the ranking stays consistent (but you wouldn't prefer preact to cycle.js v7 for having a 0.01 better result in the average slow down, would you? For ember, mithril and cycle.js v6 performance might be something to consider depending on your use case.).
If you look at the console output this is just an approximation, the real measurement is performed on data from chrome's timeline using selenium. The console measurement uses a timer to guess when rendering was performed. Of course this is more inaccurate than using the timeline.
In another words, I'd be interested more on the worst possible performance of a given framework, than their best (assuming the app is written in a reasonably good way otherwise).
Also, +1 to the sibling post that asks for longer runs, that would solve the GC question by including it altogether!
I'll keep that in mind for the next round. Let's see if that works nicely. Would you still drop outliers?
I've worked on a few apps that feature large tables like this benchmark does. With Angular for example, it's a really bad idea for performance to create DOM elements for each table row when most aren't even visible which I think is done in the benchmark:
There's lots of grid/table libraries available that use what's called virtualisation to stop lots of rows slowing the app down; rows that aren't on the screen aren't computed or rendered.
My feeling is if you avoid rendering data excessively in this way the performance differences between most frameworks aren't that important. If you're required to display 1000s of dynamic values at a time your UI probably needs work.
copy/paste into chrome console
I wish the implementers would go ahead and do a full rewrite, but there are probably too many entrenched users who would have a fit if they did...
That said, this kind of third-party benchmarking is probably useful, because so many frameworks have benchmark charts on their own site which indicate they're all faster than everything else. (Which may not even be inaccurate...for the things they benchmark, they may be the fastest. Which again maybe indicates they're all "fast enough" and unless your use case exactly matches that of the specific benchmarks in question, the benchmarks might be misleading you about which one will be fastest for your app.)
Whenever I see something like that, I usually lose interest. I don't want to see benchmarks, I want to see what it brings to the table.
- ours is X times smaller compared to Y
- ours has the full package compared to X
- you don't have to worry about doing X. You would, if you were using Y
- ours is faster than X (some graph where they're winning)
- rant on X
- we took this approach
- we enforce X as it helped us achieve Y
- we took features we liked from X, Y, Z...and came up with this
Angular and Angular-ish (I use the term loosely) frameworks like Aurelia.
Observable based frameworks (Cyclejs, Angular 2?)
React and React near-dropin replacements (Preact, React, Inferno, etc)
A bunch of MVC/MVVM frameworks that are not Angular-like (Mithril, Vue, Ember...). This is where there's the most variety, namely because many of these are a bit older (even though many kept up with time).
So there's not really nearly as many as you'd think, and in some cases migrating from one to the other as need be isn't crazy.
FYI Riot's alpha 3.0.0 release is much faster than 2.5.0, and is almost production ready (in fact, I'm using it on a project and it's rather reliable).
A LOC or character count for each benchmark would help add a bit of context.
At least it's the impression I get.
Frameworks are those that force their own opinion/style on your code structure and architecture.
Knockout is pretty much feature stable with most stuff in place, so it won't have as frequent releases as younger frameworks.
Heh, so a bunch of unreleased beta-frameworks don't count as modern any more?
Take a look at the sample implementations here:
I'd like to collaborate with others, so we don't end up with 3 different Knockout implementations ;-)
edit: My hunch was correct, see here https://github.com/facebook/react/issues/7227