
A Real-World WebAssembly Benchmark - MartinMond
https://pspdfkit.com/blog/2018/a-real-world-webassembly-benchmark/
======
bjpirt
Personal WASM anecdote: I just implemented a WASM based 3d model boolean
operation for three.js where I can intersect two models together. The pure js
version ran in around 10s on a relatively simple model whereas the WASM
version did the same thing in a little under 1s. I've been very impressed by
the technology so far! Obviously this kind of CPU intensive work is where it
can shine and it's a useful tool to have in your toolkit.

~~~
rasz
and whats the performance using native code(c/c++)? Im guessing ~0.1sec on
same models.

~~~
RussianCow
From benchmarks I've seen elsewhere, native code is generally roughly 2x
faster than WASM, so I doubt that's the case. I'm sure it depends on a lot of
factors, though.

~~~
flohofwoe
Single-threaded, non-SIMD x86-64 code being 1.2x to 2x faster than WASM is
what I'm seeing as well. Interestingly I don't see such big differences
between browsers as the PSPDFKit benchmark, my suspicion is that this mainly
measures overhead when crossing the WASM-JS border.

------
MartinMond
PSPDFKit CTO here: We've been using WebAssembly since 2017 to render PDF
documents directly in the browser and strongly believe in the future of the
technology.

We published this benchmark to have a public indicator of the performance
gains WebAssembly promises in a real-world product.

~~~
mankash666
Hello Martin - slightly off topic. There's no pricing section on your website.
Does one _have_ to get in touch with sales for a quote?

~~~
MartinMond
Yes, please contact our sales team to discuss your use case. We sell to indie
developers, early startups up to large Fortune 100 companies with a wide range
of use cases so we found custom pricing to be the best way for us to be able
to sell our software to everyone and at the same time support our team size.

------
markdog12
Firefox crushing the competition on this (all on iMac late 2015):

    
    
      Firefox Nightly: 2216  
      Chrome Canary: 5078  
      Safari Tech Preview 60: 7216
    

Brand new iPad 2018: 16653

Moto G4 phone Chrome: 37735

Windows 10 Laptop:

    
    
      Firefox Nightly: 2278
      Chrome Canary: 4907
      Edge: 16332

~~~
titzer
A big part of the score is compilation time, which improves a lot with
Liftoff, our baseline compiler for WebAssembly in V8. We've been in contact
with the benchmark authors and have identified other bottlenecks in our engine
(e.g. instantiation time), which we are now working on optimizing. Look
forward to some increasing scores soon :)

~~~
reitzensteinm
Liftoff is still substantially slower at initial compilation than Firefox. Are
there any plans to close the gap? A Unity exported game takes over 40 seconds
for a new model low end Chromebook to compile. The upcoming 3 fold decrease
will be nice, but FF could do it in 2 seconds on the same hardware.

[https://lukewagner.github.io/test-tanks-compile-
time/](https://lukewagner.github.io/test-tanks-compile-time/)

FF 61: 78.4 MB/s

Chrome Canary: 9.7 MB/s

Chrome 67: 3.3 MB/s

~~~
titzer
We still have a bottleneck at instantation time where we compile many
JS<->WASM wrappers. We are working on parallelizing this compilation as well,
so we'll get another bump out of that.

------
ebikelaw
OR I can just read and edit PDFs in the peak-optimized, ahead-of-time-compiled
native PDF reader that comes on my computer.

~~~
steipete
Ha! Not sure if you've seen it, but we offer native SDKs for iOS, Android,
Windows, macOS and Linux. Our Web SDK also has a version that uses PSPDFKit
Server via a Docker container to get faster performance.

The performance difference to native is smaller than you might think though -
see our post from a few months back:
[https://pspdfkit.com/blog/2017/webassembly-a-new-
hope/](https://pspdfkit.com/blog/2017/webassembly-a-new-hope/)

PSPDFKit for Web Standalone is a great choice for an easy-to-deploy web
viewer, annotation editor and form filler, and performance is great.
Definitely better than pdf.js - which is typically where people start with,
before they upgrade to PSPDFKit for Web.

~~~
ori_b
> Our Web SDK also has a version that uses PSPDFKit Server via a Docker
> container to get faster performance.

...what. why would I want that over a library that I can do whatever I want
with?

------
tokyodude
FYI: WebAssembly is effectively starting to get WebAssembly only APIs. Chrome
is shipping the WebGL commit API which is designed to support spin loops in
native apps (others are expected to ship it too)

Most native OpenGL apps run in a spin loop as in

    
    
        while(true) {
          update();
          render();
          glSwapBuffers();
        }
    

To support that model TPTB added a worker only `gl.commit` function that does
effectively the same thing as `glSwapBuffers` so that native apps can keep
their spin loops and don't have to refactor to be event based.

Of course if you decide to use a `gl.commit` spin loop your worker can no
longer receive events (no onmesagge, no XHR, no websockets, no fetch) so
you're supposed to use SharedArrayBuffers to communicate with a spin loop
worker.

SharedArrayBuffers are being re-enabled once they are isolated to same domain
only processes.

AFAICT this is the first browser API that is really targeted exclusively at
WebAssembly. I'm guessing more will come. Not entirely sure how I feel about
that but I guess it's inevitable.

~~~
markdog12
Are you sure gl.commit is WebAssembly only? That doesn’t seem right to me, but
I could be wrong: [https://hacks.mozilla.org/2016/01/webgl-off-the-main-
thread/](https://hacks.mozilla.org/2016/01/webgl-off-the-main-thread/)

What is TPTB?

SIMD is another WebAssembly only API, hopefully coming soon.

~~~
greggman
By WebAssembly only I don't mean you can't call it from JS. I mean rather it's
designed for Wasm not for JS. JS is generally event oriented. Native C/C++
games are usually spinloop+polling oriented. With the current design of commit
is for spinloops but once you do a spinloop you can no longer receive events.

That blog post you linked to is 2 years old and out of date with what browser
are actually planning on shipping.

For JS you no longer need commit at all. canvas (both 2d and webgl) work
offsrcreen just like they do without offscreen. You render, when your event
exits the changes get magically propagated to the corresponding canvas. commit
is solely for when your event never exits (you're in a spinloop)

------
KaoruAoiShiho
> The team was especially helpful in pointing out bottlenecks in our
> JavaScript implementation, and we’re already incorporating these changes
> into our next PSPDFKit for Web release.

Any chance you can talk about those bottlenecks? Might be interesting.

------
stcredzero
We should be at the point where network computers are practical for casual
users. A full install of the BeOS operating system is just 45MB. A tricked-out
one was barely more than a couple of hundred MB. We should now be able to have
operating systems which are mostly cached from online connections, running in
clients like Web Browsers, but purpose built for running such an OS.

------
symboltoproc
What's the difference to benchmarks like
[https://hacks.mozilla.org/2018/01/oxidizing-source-maps-
with...](https://hacks.mozilla.org/2018/01/oxidizing-source-maps-with-rust-
and-webassembly/)? The results seem vastly different.

~~~
the_mitsuhiko
One is a large c++ compiled ti asm.js and wasm. The other is wasm compiled
rust vs javascript. How are they comparable?

~~~
sp332
GP is just wondering why the results are so different.

~~~
steipete
You can't compare small/micro-benchmarks that call into WebAssembly and then
get the result with a larger full-stack application where parts are in JS and
parts in WASM - both are highly interlinked. This is much more reflective of
real products. We couldn't find a benchmark that shows such end-to-end usage,
so we built one.

------
singularity2001
the standalone version produces a dom tree?

I expected canvas rendering, which would be MUCH faster. (?)

<div style="top: 628.75px; left: 125px; position: absolute; transform-origin:
0px 0px 0px;"><label style="width: 38px; height: 38px;"
class="PSPDFKit-43pb73xrbckjuggksebahxjw9 PSPDFKit-Annotation-Widget PSPDFKit-
Annotation-Widget-CheckBox"><input name="Checkboxes 2" value="Checkboxes"
type="checkbox"><span class="PSPDFKit-v4b9qprfhz2hybx1p43s4xgx6
PSPDFKit-8vpzte5r65p8pf9q4d61qcb34u" style="border-width:
1.25px;"></span></label></div>

~~~
josephg
If you render to canvas, you would need to reimplement scrolling and zooming,
checkboxes and input elements. Any custom reimplementation of this stuff will
end up slower and clunkier than the optimised, native versions already in
browsers.

Also (as far as I know) canvas content is inaccessible to screen readers.
Accessibility matters.

~~~
pjmlp
No problem.

[http://example.qt.io/qt-
webassembly/quickcontrols2/gallery/g...](http://example.qt.io/qt-
webassembly/quickcontrols2/gallery/gallery.html)

------
Matthias247
> Browser vendors can reach out to us and obtain a more permissive license key
> so that the benchmark can run on different machines and even on their
> continuous integration servers.

Sounds like a great idea for encouraging performance improvements and avoiding
regressions!

------
yandrypozo
wondering why those benchmarks weren't run on Linux distros like Ubuntu,
Debian, etc.

------
billybolton
Now all we need is to expose a graphics context to wasm. No need for DOM
support.

~~~
markdog12
You already get 2D, WebGL, WebGL 2 through canvas

------
aioprisan
Great work, looks like a very useful PDF toolkit. I've been working on a side
project implementing something similar for some time but has stalled due to
life getting in the way. Can you share anything publicly in terms of pricing?

I see everything is through a sales person now and I'd like to get a better
high level understanding of high level costs for passion projects (<1000
users), small-medium (<100k), and enterprise (1M+). What metrics are you
charging usage on?

~~~
steipete
Metrics are required platforms (iOS, Android, Web, Windows, macOS, Linux),
components (see matrix on [https://pspdfkit.com/pdf-
sdk/](https://pspdfkit.com/pdf-sdk/)), number of users, number of server nodes
(if Docker-based), number of apps/[sub]domains, internal/external access and
expected support, to name a few. We're usually quite fast to figure something
out that works for both parties, pricing is in the 4 (passion projects) to 6
(enterprise) figure range. Simply drop us a message via
[https://pspdfkit.com/sales](https://pspdfkit.com/sales) to get things
started. We try to make this process as easy as possible.

~~~
mkl
You should probably consider putting something basically exactly like this
paragraph on your site, especially the pricing range.

~~~
aioprisan
^ this. Please provide a pricing page with a few options and even a usage
calculator. Without it, you'll be hard pressed to find enough folks giving
this a real shot and will just assume that it's enterprise-only and out of
their range.

------
krzyzanowskim
Safari is so slow

~~~
swaroop
Looks like they are working on it, which is good news ->

From
[https://twitter.com/steipete/status/1013035117076467713](https://twitter.com/steipete/status/1013035117076467713)
:

Shoutout to the Safari team! They were able to track down the slowness in the
current WebAssembly implementation and ALREADY FIXED IT. Patch is here:
[https://trac.webkit.org/changeset/233378/webkit](https://trac.webkit.org/changeset/233378/webkit)

(trace points caused a significant (4x!) slowdown as our benchmark performs
many calls into the VM)

------
thosakwe
One thing that would be super nice, in theory, would be compiling WASM modules
on-the-fly from the text format.

In a sense, it would be like a JIT for the Web, and one possibility for
bringing existing JIT-based languages to WASM.

~~~
thosakwe
In case anyone else was wondering, you actually _CAN_ do this and JIT WASM in
your browser via Binaryen:

[https://github.com/WebAssembly/binaryen/blob/master/test/bin...](https://github.com/WebAssembly/binaryen/blob/master/test/binaryen.js/hello-
world.js)

------
KaoruAoiShiho
What is the most performant compile to wasm language? Or are they all similar?

~~~
timerol
Since all languages should be able to compile to wasm, that question is pretty
close to "Which language is the most performant?", which is a can of worms
that I'm not going to open. Maybe something like the Benchmarks Game would be
useful here. Or already exists, and someone could point me to it :)

A similar question would be "What languages do you know of that are focused on
performance and compiling to wasm?" For that, I'd say C++, Rust, and Go,
though I'm sure there are others.

------
bigato
> In the future, Microsoft wants to bring more features — which are currently
> not possible in asm.js (or JavaScript) — to WebAssembly.

This is very vague, but already disturbing

~~~
mtgx
I'm guessing these "features" will be tied to the Microsoft Store, something
they're already doing with PWAs.

Many were quick to criticize Valve and Epic about their own criticism of the
Microsoft Store, which is why Microsoft is now using a "boiling the frog"
strategy to tie everything to the Store over the next decade or so. And this
process will likely accelerate once Windows 7 is dead and buried.

~~~
pjmlp
For those that don't know it, WinRT always supported hosted web apps.

Microsoft has picked up on Google's PWA ideas and is migrating UWP hosted web
apps to it.

I expect Google to follow a similar path with Android and ChromeOS.

------
mrmondo
\- Firefox 62b5: 1429

\- Chromium 68: 5722

\- Safari 11.1.1: 6500

On 2017 Macbook pro running macOS 12.13.5 on battery power

------
owaislone
I tried on Ubuntu 18.04. Firefox around 2k and Chrome around 10k

