
Massive: The asm.js Benchmark - rnyman
https://hacks.mozilla.org/2014/11/massive-the-asm-js-benchmark/
======
higherpurpose
I prefer asm.js over NaCl, especially when NaCl has been available for years,
and Google still supports the ARM architecture poorly, and as a second class
citizen in NaCl.

It's unacceptable that a "browser-os" that should have no problem being
architecture agnostic, _still_ gives Intel an edge with ChromeOS, because not
all NaCl apps work on ARM Chromebooks. Seriously, how _crazy_ is that?

I can understand ARM not having a serious chance on Windows laptops because of
all the legacy x86 programs, but at worst it should be equal with x86 running
an OS such as Chrome OS.

~~~
TazeTSchnitzel
I agree. asm.js has several strengths over NaCl and even PNaCl:

* It is just JavaScript, so any browser can execute asm.js code, but it can be AOT compiled by some browsers for extra performance - This means _every_ browser supports asm.js, and supports it today

* It doesn't require a new runtime or API, merely using the existing web APIs

* It is simple for other browser vendors to implement, and isn't reliant on a single implementation

* It can interface with existing JavaScript code, so it can use JS libraries and, similarly, JS code can use asm.js libraries

* It is a completely open standard

* It is architecture-independent (so's PNaCl, though)

~~~
saosebastiao
> It is just JavaScript, so any browser can execute asm.js code, but it can be
> AOT compiled by some browsers for extra performance - This means every
> browser supports asm.js, and supports it today

I feel this point is slowly becoming disingenuous. If someone goes through the
trouble of compiling their application to asm.js, they obviously really care
about performance, because "normal" javascript wasn't good enough for them.
Hell, elm/cljs/scala.js/js_of_ocaml/etc wasn't good enough. How many asm.js
developers do you know that will go through the trouble of compiling a
codebase to asm.js, without putting up an alert('This app works best on
Firefox');?

Unless the barriers to entry for compiling to asm.js can be _very
significantly_ reduced, it _will become_ a Mozilla-only ecosystem.

I'm a huge Mozilla fanboy, but I feel like they bit off more than they could
chew here, and I think the effort would be better spent on standardizing a
spec for a true VM instead of trying to back-compat-hack one into javascript.

~~~
TazeTSchnitzel
> I feel this point is slowly becoming disingenuous. If someone goes through
> the trouble of compiling their application to asm.js, they obviously really
> care about performance, because "normal" javascript wasn't good enough for
> them.

...what? asm.js is a compiler target for C code. It's not intended to be hand-
written. It's also the _default_ compiler target for C code on the web. You
wouldn't really want to compile to anything else, except NaCl I suppose
(though that requires more porting effort).

I don't get what you mean by "going through the trouble of compiling their
application to asm.js" versus ""normal" javascript wasn't good enough for
them.". Why would they compile to normal JS? There's no reason to: Their
codebase is in C, C++, or some other native code language. If they compile to
asm.js, they get a free performance boost. If they compile to JS, they just
get horrible slowness. The only difference between them, here, is a single
compiler flag. I don't understand.

> Hell, elm/cljs/scala.js/js_of_ocaml/etc wasn't good enough. How many asm.js
> developers do you know that will go through the trouble of compiling a
> codebase to asm.js, without putting up an alert('This app works best on
> Firefox');?

Why would normal JS developers be writing in asm.js? Do you even understand
why asm.js exists?

> Unless the barriers to entry for compiling to asm.js can be very
> significantly reduced, it will become a Mozilla-only ecosystem.

What barriers to entry? You download the emscripten SDK and you compile your
existing C/C++ app to asm.js.

> I'm a huge Mozilla fanboy, but I feel like they bit off more than they could
> chew here, and I think the effort would be better spent on standardizing a
> spec for a true VM instead of trying to back-compat-hack one into
> javascript.

It's not a VM, there's no JIT involved.

~~~
saosebastiao
If Opera/Dolphin/Safari/whatever don't ever implement optimizations for
asm.js, its not really cross-platform anymore. Sure, theoretically they can
execute the code, but the performance will be so far from the expectations of
the developer that they will just end up telling the user to switch to Firefox
(maybe Chrome if they can ever catch up).

> Why would normal JS developers be writing in asm.js? Do you even understand
> why asm.js exists?

Normal JS developers would use asm.js if they needed extra performance. I
occasionally need extra performance out of javascript, so I use heavily
optimized clojurescript, which helps for some applications. If I had exhausted
those options, I would try programming in C and compiling to asm.js.

> What barriers to entry? You download the emscripten SDK and you compile your
> existing C/C++ app to asm.js.

Ha. You obviously haven't tried to do that yet. asm.js is intended to be a
compiler target for many languages. It only succeeds for two languages
(C/C++), thanks to emscriptem more than asm.js, and even that is not trivial.
It takes significant effort to port most C/C++ applications...otherwise they
wouldn't have their own codebases, they would just be a different target on a
makefile. If it were as simple as you make it out to be, we would have
hundreds of languages compiling to asm.js by now; C/C++, but also every
language that compiles to C/C++ or LLVM bytecode. As it is, we have maybe a
couple dozen large legacy codebases that have been ported.

> It's not a VM, there's no JIT involved.

A VM does not necessarily imply JIT, but that is beside the point. A VM
executes bytecode, and asm.js is an attempt to use javascript as bytecode.
Javascript wasn't designed as a VM, and several of its "features" actively
subvert its use as a VM, which is why asm.js relies on guarantees that you
will not use those features.

If getting the performance that asm.js developers expect out of asm.js
requires the use of a browser that optimizes for asm.js, then the solution
isn't cross-platform. In fact, it isn't really any better than NaCL, god
forbid.

~~~
TazeTSchnitzel
> If Opera/Dolphin/Safari/whatever don't ever implement optimizations for
> asm.js, its not really cross-platform anymore. Sure, theoretically they can
> execute the code, but the performance will be so far from the expectations
> of the developer that they will just end up telling the user to switch to
> Firefox (maybe Chrome if they can ever catch up).

I'm not sure that's really true. It doesn't have to be ultra-fast, just good
enough.

> If it were as simple as you make it out to be, we would have hundreds of
> languages compiling to asm.js by now; C/C++, but also every language that
> compiles to C/C++ or LLVM bytecode. As it is, we have maybe a couple dozen
> large legacy codebases that have been ported.

Er, well, there aren't many languages which compile to native code. People
can't compile Java, C#, JavaScript, Perl, Python, Ruby, PHP, etc. to native
code. C/C++ do compile to native code, however.

> Ha. You obviously haven't tried to do that yet.

Oh, I have. I realise it has some difficulty to it.

> If getting the performance that asm.js developers expect out of asm.js
> requires the use of a browser that optimizes for asm.js, then the solution
> isn't cross-platform. In fact, it isn't really any better than NaCL, god
> forbid.

I'm not sure what you think asm.js is the solution to. Its purpose is for
running native applications in the browser. It's not for making ordinary JS
apps run fast.

------
StevePerkins
I'm often dismissive of Mozilla, for being so all-over-the-map and unfocused
sometimes. However, I'm grateful for them being the only browser player that
really does seem interested in progressing JavaScript.

All of the other players seem more focused on transpiling solutions, which are
really about replacing JavaScript with something else under their control.
Mozilla is pretty much the only player that genuinely works to improve
JavaScript without another agenda.

~~~
cromwellian
asmjs is a transpiling solution. No one is going to rewrite UnityEngine in
hand-written asmjs. The vast majority of asmjs code is going to be transpiled,
you can literally see it in the name: _asm_

~~~
TazeTSchnitzel
Yes, it is a transpiling solution, but it's one that's a subset of JavaScript.
This means it's compatible with all JS engines and lives in the same
ecosystem.

It's quite different from, say, NaCl or Dart, which introduce a completely new
language with a new set of APIs.

~~~
cromwellian
Dart the platform is compatible with all JS engines, Dart2JS.

I could rightly claim that DartVM is just a specialized two-stage compiler
that kicks in to give a speed boost.

Compatible != portable. In the real world, performance matters. If I deploy my
new triple-A game to the Web, and I'm only going to get 60fps on 25% of the
browsers, and 15fps on the rest of them, do I really consider it the same
ecosystem?

~~~
azakai
> If I deploy my new triple-A game to the Web, and I'm only going to get 60fps
> on 25% of the browsers, and 15fps on the rest of them, do I really consider
> it the same ecosystem?

This was always an issue on the web. Browsers have always had speed
differences on that scale - either due to browser optimizations, or running on
much weaker hardware (a phone vs. a powerful desktop). We've never had
consistency the way consoles do, which is why AAA games run best there.

------
amelius
They forgot to include two things:

    
    
      * Performance on the native architecture
      * Performance on other browsers (Chrome/IE in particular)

------
itsbits
would love to know how is the performance in V8 chrome js engine??

~~~
acqq
Run it with your Chrome and post the results:

[http://kripken.github.io/Massive/](http://kripken.github.io/Massive/)

~~~
fridek
I have Intel i7 running on Linux.

Got 2,984 running Chrome 38 and 8,778 on Firefox 33, but I think now it's just
fair to wait for couple of iterations of Chrome to identify bottlenecks and
fix them.

It's not uncommon for benchmarks to focus on some particular strength/weakness
that is easily fixable. The same happened with Firefox and Octane 2.0.

~~~
TazeTSchnitzel
I'm not sure if Chrome is ever going to catch up to Firefox here. It's been
slower than Firefox at asm.js ever since it was announced, because Chrome
doesn't do AOT compilation for it.

EDIT: Although I suppose JIT compilers would love asm.js as it has all the
type info they want, but there's probably tons of overhead there thus
explaining how Chrome has worse results.

~~~
pjmlp
I would say Google team has other priorities....

~~~
TazeTSchnitzel
Yes, like trying to destroy the open web with NaCl.

~~~
blibble
I don't really understand how asm.js is any different...

you've still got applications compiled into binary blob, the difference
between NaCl and asm.js being that asm.js just happens to be binary encoded as
executable JavaScript.

~~~
TazeTSchnitzel
asm.js isn't a binary blob, it's JavaScript. It's not a threat to a web of
open standards.

~~~
cromwellian
It's a blob transpiled from C, not exactly how people imagined the Web working
a few years ago. It's a non-W3C/TC39 approved set of compiler
intrinsics/pragmas that switches on special behavior in one particular
environment.

When Dart was introduced years ago, Brendan Eich was all over HN complaining
about how Dart2JS doesn't solve the problem, because DartVM would be more
performance, and introduce a tendency to optimize JS VMs towards those
semantics. Well, here we have a different problem. If asmjs takes off and
everyone starts writing Web apps by transpiling C code, Mozilla will have
forced a sea change in the Web, forcing all other vendors into ahead-of-time
compilation to compete, and forcing JS VMs to optimize for this particular
kind of code.

Now, maybe that's not such a bad thing. I'm not arguing it isn't, but I think
arguing "it's just JS, the syntax is the same, therefore it's open!" is
somewhat disingenuous. ASMJS, if super successful, would create an implicit
implementation spec for everyone. It will have in essence, created a language
with identical syntax to Javascript, but with completely different expected
execution semantics.

Why? Because developers would come to _depend_ on ahead-of-time compilation,
depend on apps that basically use emulated malloc'ed heaps, etc.

Imagine I took the Java language, removed "new" and forced all code to
allocate DirectBuffers, and then said, all classes must consist of static
final methods (no polymorphism), and can't have any fields or constructors.

Call this ASMJAVA. I now assert, this is "Just java", hey, it runs in any JVM.
Except, I'm shipping a new VM called "SuperExpresso" that does ahead-of-time
compilation for my mobile platform based on this, and I'm telling people they
don't need to use native SDKs, they can just transpile C code to Java. Did I
create a new standard? I think I just did.

Arguably, you can say the same thing about Android 4.4/5.0 "ART" runtime, or
GWT which I work on. They use the Java "language", but the runtime behavior is
completely different, and at least for GWT, we make no bones about this.

So, this is not an attack on asmjs, but I do think there is a distinction
between "reusing the syntax" of a language, and creating expectations of new
semantics. Languages carry with them implicit sets of runtime performance
characteristics, even bugs, that developers tend to treat holistically.

Not many people care about what the specs say. They care about how it runs in
the real world.

~~~
TazeTSchnitzel
> It's a blob transpiled from C, not exactly how people imagined the Web
> working a few years ago.

It's not readable, nicely-formatted JS, sure, but neither is the optimised,
minified JS most sites served. It is, however, valid JavaScript. It uses open
standards (ECMAScript, the various web APIs). It's already supported by all
browsers.

> It's a non-W3C/TC39 approved set of compiler intrinsics/pragmas that
> switches on special behavior in one particular environment.

It's not "special behaviour". It's a performance optimisation. It still
behaves exactly like JavaScript.

> If asmjs takes off and everyone starts writing Web apps by transpiling C
> code, Mozilla will have forced a sea change in the Web, forcing all other
> vendors into ahead-of-time compilation to compete, and forcing JS VMs to
> optimize for this particular kind of code.

I don't see why people would write Web apps by transpiling C code, the main
advantage is for the existing codebases written in C. And other vendors don't
_necessarily_ need to implement asm.js with AOT compilation. asm.js-conformant
code works well very with JIT compilers.

> Now, maybe that's not such a bad thing. I'm not arguing it isn't, but I
> think arguing "it's just JS, the syntax is the same, therefore it's open!"
> is somewhat disingenuous. ASMJS, if super successful, would create an
> implicit implementation spec for everyone. It will have in essence, created
> a language with identical syntax to Javascript, but with completely
> different expected execution semantics.

No, it has identical execution semantics. It uses a static subset of JS, yes,
but it's still JavaScript. If I remove the _" use asm"_ pragma, the code still
does _exactly_ the same thing.

> Why? Because developers would come to depend on ahead-of-time compilation,
> depend on apps that basically use emulated malloc'ed heaps, etc.

Is that bad? Bear in mind that, without asm.js existing, this was the case
anyway. asm.js wasn't created so that people could compile native code to run
in the browser. It was created because people were already doing so, and by
detecting the code patterns commonly used by compilers, they could make it run
faster.

> Call this ASMJAVA. I now assert, this is "Just java", hey, it runs in any
> JVM. Except, I'm shipping a new VM called "SuperExpresso" that does ahead-
> of-time compilation for my mobile platform based on this, and I'm telling
> people they don't need to use native SDKs, they can just transpile C code to
> Java. Did I create a new standard? I think I just did.

Sure, you made a new standard. It's a backwards-compatible one that doesn't
rupture the ecosystem, though.

