
Perspective: Streaming pivot visualization via WebAssembly - texodus
https://github.com/jpmorganchase/perspective
======
dman
I wrote the C++ part of this codebase, would be happy to answer any questions
about it here.

~~~
pavlov
Is Emscripten the only game in town for WebAssembly C/C++ toolchains, or are
there other contenders?

~~~
Klathmon
Even outside of C/C++, Emscripten seems to be required in some fashion for
just about every "compile X to webasm". Which i'm not that happy about as it's
quite heavy and takes a long time to compile, which makes it harder for
newcomers to try it out because it requires a fairly large time investment to
even compile hello world.

~~~
ekidd
> Even outside of C/C++, Emscripten seems to be required in some fashion for
> just about every "compile X to webasm".

One exception to this is Rust's new "wasm32-unknown-unknown" target, which
uses LLVM to directly generate wasm files without going through emscripten:
[https://www.hellorust.com/news/native-wasm-
target.html](https://www.hellorust.com/news/native-wasm-target.html)

The easiest way to use it for larger projects is probably via
[https://github.com/koute/cargo-web](https://github.com/koute/cargo-web),
which handles a lot of packaging very nicely.

It's really nice to use, assuming you already know Rust.

~~~
Klathmon
Yeah I'm super excited about Rust's new webasm compiler, as well as some new-
ish stuff with .Net that works without Emscripten.

I get the ecosystem is super young, but it was honestly what kept me from
playing with Webasm for a while, because each time I'd sit down to play with
it I'd be in for a 3 hour compile after spending an hour getting my windows
machine to have the right compilers and not stomping over other things I have
setup on here.

~~~
azakai
Sounds like you were seeing an old emscripten SDK bug, where it compiled
LLVM+clang unnecessarily and with debug info (which is very slow).

Currently there shouldn't be anything like that, it will download a binary
build of LLVM+clang for your system. It should be ready to use immediately
after that download.

------
mi100hael
Really cool to see non-software companies putting cool projects like this out
as open source

~~~
ashtonian
the stock market is digital, they are a software company.

~~~
lotyrin
Having worked at a branch off a university devoted to its online school and
content marketing where the inputs and outputs of production are entirely
digital, I can say they sure as hell aren't a software company; nobody above
on-the-ground grunts think much about even the big pictures of domain modeling
/ workflows, data ingestion / transformation, computing / efficiency /
performance, software development lifecycle / product portfolio management /
QA, growth hacking / optimization / testing etc. because they simply aren't
interested in the slightest. I'm sure plenty of organizations with at least
one foot in pre-internet industries are in the same boat, and it's exciting to
see counterexamples.

~~~
jnordwick
I think it is more correct to say that a large subunit of JPM works as a
software company -- for its internal use. All modern financial companies are
heavily tech oriented, even those you wouldn't normally think of that way. The
best tech firms I've ever been involved with financial firms from front desk
trading on back. One of the best tech teams I've ever met with has been one of
Bloomberg's teams dealing with their internal core services and bond platform
(I was totally floored at how amazing these guys were). These companies are
too large to say they are entirely software, but there are definitely
Directors and CTOs that live and breath all those things you mention.

~~~
lotyrin
Sure yes, in modern business environments, everyone's business interacts at
least in some way with computing, digital products like media and software,
and the internet. Even if you had a remote industrial site cutoff from the
wider internet, there would be some automation to be done and some code to do
it. Every organization should have a CTO, if nothing else than to make
procurement and outsourcing decisions. Organizations shouldn't outsource that
which differentiates them, and there's lots of room for differentiation in the
quality and degree of automation, thus internal software teams. Etc.

All I'm getting at is that those present set of truths, that that reality
hasn't spread everywhere yet, and it's still worth celebrating when it has and
when it goes well.

There are plenty of people given plenty of power in leadership that won't
bring themselves to understand technology and have yet to retire.

~~~
btown
You would be pleasantly surprised, then, that this reality has already
permeated the modern financial industry. For instance, see
[http://www.businessinsider.com/goldman-sachs-wants-to-
become...](http://www.businessinsider.com/goldman-sachs-wants-to-become-the-
google-of-wall-street-2017-4) . While there are still “relationship bankers”
in expensive suits, most acknowledge that software is key to giving them an
edge over the competition, and that building a culture around software only
helps them.

------
bpicolo
This is great. Finance people especially love this sort of thing - they do
want to browse through entire datasets vs seeing higher level metrics in a lot
of cases.

~~~
JumpCrisscross
> _Finance people especially love this sort of thing - they do want to browse
> through entire datasets vs seeing higher level metrics in a lot of cases_

To expand on this, a lot of finance is searching out and understanding,
normalizing or exploiting edge cases.

~~~
testnet
This is why excel is king. just give me a grid with all the data and a pivot
table

~~~
rch
Except when Excel won't manage nearly all the data. I'll sometimes use Igor
Pro† before coding something up however.

† (only because I'm familiar with it - I'm positive there are better GUIs)

------
ChicagoDave
I love seeing new UX on Web Assembly. Someone needs to port or create a
standard layout engine like WPF to build on this. If the tools were there, C#
+ VS Code + solid layout engine...I'd dump JS + HTML + CSS in a "flash".

~~~
pjmlp
There are some attempts on that direction.

[https://www.hanselman.com/blog/NETAndWebAssemblyIsThisTheFut...](https://www.hanselman.com/blog/NETAndWebAssemblyIsThisTheFutureOfTheFrontend.aspx)

~~~
ChicagoDave
Sure. I've played with Blazor. It's very raw at this point and just a side
project. I'd like to see MS Research or even a full blown R&D team at MS go
full force at this.

~~~
pjmlp
Microsoft just announced that they will start investing more into it,
depending how the community takes it.

[https://blogs.msdn.microsoft.com/webdev/2018/02/06/blazor-
ex...](https://blogs.msdn.microsoft.com/webdev/2018/02/06/blazor-experimental-
project/)

------
elwell
"J.P. Morgan open sources..." \-- that's so cool

------
polskibus
Amazing, despite the fact that some heavy-lifting is done by highcharts and
hypergriid.

What's missing for this to effectively kill all typical BI tools like PowerBI,
Tableau?

~~~
infinite8s
Expensive sales teams. Also neither of those handles streaming data.

------
andrewmcwatters
@dman Cool, but I'm curious why WebAssembly was chosen for this project. What
was the value of using it here versus just using the Web API?

~~~
dman
The following features made wasm compelling here

a. Filter / Pivot / Aggregate moving clientside means a lot of load off the
servers.

b. Performance in wasm is surprisingly close to same C++ code running natively
on the desktop.

c. Ability to reuse existing C++ codebase and use it to build both native and
web apps.

------
bcherny
Presumably, the point of using WASM and service workers is for performance,
but I see no benchmarks. I also have trouble imagining that this actually
improves performance, unless you’re doing all of your compute in the browser
and it’s CPU bound (this seems like bad design).

What is the performance of doing it this way vs. without WASM and workers?

Until we see numbers, this smells like a recruiting play.

~~~
lucozade
For ticking data being pushed into bespoke pivots, running this on the backend
and, say, pushing renders to the frontend isn't generally better. You're not
likely to be sharing much of the grunt. Also, latency will be an issue so you
need to be pretty careful with GC runtimes that aren't optimised for latency.

> What is the performance of doing it this way vs. without WASM and workers?

@dman says in another thread that the original was in Python and this was 10x
faster. If this met their performance goals, it does the job. It may be
possible that another way is better, for some definition of better, but who
cares if it works as required?

> Until we see numbers, this smells like a recruiting play.

It's presumably OK not to use it if you don't want to.

And what if they think this may help recruiting? Seems like a reasonable trade
to me.

------
adamsea
First, let me say this looks awesome! I’m curious if your considered React.js
for the presentation layer (i.e. the js/html5 parts) - on the surface it seems
like it would be a good fit - and if so, what you saw as the benefits or
drawbacks of it vs the approach you went with.

~~~
texodus
We chose to go with a Web Components based interface for compatibility across
frameworks, but alot of where we go in the future will be determined by the
expertise of the developers we hire to work on it, and the community if there
is interest there.

~~~
drglitch
First of all, great work everyone involved! I know first-hand how hard it is
to push something like this past compliance ;)

Going with WC is a sound decision since its trivial to wrap it into ember,
react, or any other framework-du-jour... especially so since more and more of
them are adopting WC patterns and design practices.

I might try to wrap this into react and will see if bosses permit opensourcing
it!

------
hackcasual
For those interested, keep in mind the only way to use WebAssembly with CSP in
Chrome is by turning on 'unsafe-eval'. FF/Edge/Safari all at least support
compilation from URLs with more locked down policies

~~~
texodus
True - worth pointing out that this library gracefully falls back to ASM.js if
your browser does not support WebAssembly, though.

~~~
hackcasual
Does it fallback if Window.WebAssembly doesn't exist, or does it fallback if
WebAssembly.instantiateStreaming fails?

~~~
texodus
The former. In practice, it seems some runtimes are still a bit buggy - so we
also revert to ASM.js if e.g. you are on an iPhone

------
devscreen
This is seriously cool. Unfortunately I have no use-case for it in the product
I currently work on, but still fascinating to flick through the source code
and play with it.

------
mjansen
really cool. would love to see a Rust version.

~~~
shamaku
build it

------
flight21
What are you using for UI, any framework (Polymer, D3) or just plain
WebComponent?

~~~
texodus
Plain web components. The UI is quite light in terms of functionality so far.

------
polskibus
Does the C++ code use vectorization? If not, is it planned?

------
baybal2
So, what there is about visualisation?

I read your docs, and found nothing about how to actually render stuff to the
page.

------
wsgeek
I followed the directions to the letter, and everything installed correctly on
MacOS. When I tried the same on Linux, though, I got tons of the following
errors on the build step (keep in mind I do have boost development libraries
installed in /usr/include):

(Sorry about the formatting.... I tried <code>). Anyway, TL/DR it cannot find
boost.

$ ./node_modules/.bin/lerna run start --stream lerna info version 2.8.0
@jpmorganchase/perspective: > @jpmorganchase/perspective@0.1.1 start
/home/dj/usr/src/perspective-clone/packages/perspective
@jpmorganchase/perspective: > npm run compile && (npm run perspective & npm
run compile_test & npm run compile_node & wait) @jpmorganchase/perspective: >
@jpmorganchase/perspective@0.1.1 compile /home/dj/usr/src/perspective-
clone/packages/perspective @jpmorganchase/perspective: > mkdir -p build
build/wasm_async build/wasm_sync build/asmjs && (cd build/; emcmake cmake ../;
emmake make -j8; cd ..) @jpmorganchase/perspective: -- Configuring done
@jpmorganchase/perspective: -- Generating done @jpmorganchase/perspective: --
Build files have been written to: /home/dj/usr/src/perspective-
clone/packages/perspective/build @jpmorganchase/perspective: Scanning
dependencies of target psp @jpmorganchase/perspective: [ 1%] Building CXX
object CMakeFiles/psp.dir/src/cpp/base_impl_win.cpp.o
@jpmorganchase/perspective: [ 2%] Building CXX object
CMakeFiles/psp.dir/src/cpp/arg_sort.cpp.o @jpmorganchase/perspective: [ 4%]
Building CXX object CMakeFiles/psp.dir/src/cpp/calc_agg_dtype.cpp.o
@jpmorganchase/perspective: [ 5%] Building CXX object
CMakeFiles/psp.dir/src/cpp/aggspec.cpp.o @jpmorganchase/perspective: [ 8%]
Building CXX object CMakeFiles/psp.dir/src/cpp/aggregate.cpp.o
@jpmorganchase/perspective: [ 8%] Building CXX object
CMakeFiles/psp.dir/src/cpp/base.cpp.o @jpmorganchase/perspective: [ 9%]
Building CXX object CMakeFiles/psp.dir/src/cpp/base_impl_linux.cpp.o
@jpmorganchase/perspective: [ 10%] Building CXX object
CMakeFiles/psp.dir/src/cpp/build_filter.cpp.o @jpmorganchase/perspective: [
12%] Building CXX object CMakeFiles/psp.dir/src/cpp/column.cpp.o
@jpmorganchase/perspective: In file included from
/home/dj/usr/src/perspective-
clone/packages/perspective/src/cpp/calc_agg_dtype.cpp:11:
@jpmorganchase/perspective: In file included from
/home/dj/usr/src/perspective-
clone/packages/perspective/src/include/perspective/calc_agg_dtype.h:12:
@jpmorganchase/perspective: In file included from
/home/dj/usr/src/perspective-
clone/packages/perspective/src/include/perspective/schema.h:13:
@jpmorganchase/perspective: /home/dj/usr/src/perspective-
clone/packages/perspective/src/include/perspective/base.h:29:10: fatal error:
'boost/unordered_map.hpp' file not found @jpmorganchase/perspective: #include
<boost/unordered_map.hpp>\n ^~~~~~~~~~~~~~~~~~~~~~~~~ </code>

------
texodus
If anyone finds this project exciting and is interested in learning more about
working on Open Source at J.P.Morgan, feel free to send me a message - we are
always looking to hire experienced, passionate talent!

~~~
feel_the_need
I would like to work on cakeshop-- how do I get on touch with you?

~~~
patrickmn
For the blockchain projects, please email quorum_info@jpmorgan.com

------
ChicagoDave
Poor naming choice. Google Perspective has been around for a year or two I
believe.

~~~
timkpaine
in that case, its actually a good naming choice, as this existed (in a closed
source form) for much longer than that.

~~~
ChicagoDave
I'm not knocking the code...I think it's brilliant. I just think you're going
to cross wires with Google here. Will perspective mean "filtering troll
comments" or will it mean "displaying analytics in web assembly" is an
unfortunate set of choices.

~~~
dman
The name had been in place internally since 2013, so the naming ship had
sailed a long time ago.

------
kchoudhu
It's open sourced from an investment bank, so you'll never see it updated
again. Beware of integrating it into your projects.

Good on whoever took on legal to get this out the door though. Hope you make
it to MD or whatever it is you were gunning for.

~~~
texodus
We very much plan to continue developing this entirely in the open! It is
reputationally important we do not just dump our dead projects on the internet
as Open Source, which is one of the reason why we chose Perspective for this
project in the first place.

