WebAssembly can be certainly faster than JS in certain situations, but well-structured, monomorphic JS with the kind of type-hints for numeric values that asm.js is based off of can be extremely fast as well.
It'd be great to check the assumption that WASM is actually faster in this case, and by how much, especially given the friction of sending anything other than numbers across the JS/WASM boundary.
Funny enough: was in NYC today talking with banking teams about related tech. Too bad we didn't know about this effort, would have loved to meet!
We have met before actually, you did a demo at JPM in midtown several years back. Graphistry has come a very long way since then - impressive work!
* grid format
* graph format
* actual pivoting of data
Is that right?
It's using webassembly so in effect to do the pivot you must have created functions that exist in pandas?
Has anyone ported pandas to webassembly?
Is the grid editable? What are you using to create your output, html or canvas or something?
I realise I could look this stuff up but since you asked..
The grid is just a plugin for the excellent [Hypergrid](https://github.com/openfin/fin-hypergrid), which is editable, but you'd need to manually push those edits back to the engine for now.
So PyPy.js runs Python in the browser via asm.js:
Livebook uses PyPy.js to make a Jupyter-like notebook that runs entirely in the browser, and includes pandas:
Hypergrid is a canvas-based renderer, yes.
Also, what methods are available for getting data streamed [into Perspective]? I'm skimming through the docs, but am not quite getting it...
[edited for clarity]
Apologize for the documentation clarity - this is our first release and very much under active construction!
I didn't have time for a proper look yet, so no worries. If at all possible post the code and data (or something like it) that is in those animations you have. That would perhaps allow users to dive in by tinkering here and there.
Do you feel like web assembly could potentially replace JS in the future with, say, my favorite back end language of choice?
The only thing I see languages compiled to WebAssembly taking over is development of highly resource intensive applications, algorithms or libraries. Or if there's a quantum leap with some specific language + framework that eliminates a lot of work with current frontend development. Otherwise, there will be too much fragmentation and you'll see the regression to the mean effect as has been the case with predecessors such as CoffeeScript, Elm, PureScript, ReasonML, and every other compile-to-JS language du jour.
The possibility of some company creating, from scratch, a new web development paradigm, given everything we know today, is exciting.
virtualization (showing only a view of full dataset that is updated from server on scroll)
Pivoting (multiple levels/hierarchies on rows and on columns)
Apart from features what part did you consider was the biggest hurdle? What browsers did you target?
N levels of pivoting on both axes.
Should work on all browsers - if not, please open an issue!
Though, we have quite alot of experience doing this in the past - and the design of Perspective is very much a reaction to what we learned, at lesat in regards to the typical financial dataset which is much smaller than 1TB.
Worst part (which has improved dramatically) was the rough edges in the web assembly toolchains early on.
One exception to this is Rust's new "wasm32-unknown-unknown" target, which uses LLVM to directly generate wasm files without going through emscripten: https://www.hellorust.com/news/native-wasm-target.html
The easiest way to use it for larger projects is probably via https://github.com/koute/cargo-web, which handles a lot of packaging very nicely.
It's really nice to use, assuming you already know Rust.
I get the ecosystem is super young, but it was honestly what kept me from playing with Webasm for a while, because each time I'd sit down to play with it I'd be in for a 3 hour compile after spending an hour getting my windows machine to have the right compilers and not stomping over other things I have setup on here.
Currently there shouldn't be anything like that, it will download a binary build of LLVM+clang for your system. It should be ready to use immediately after that download.
Rust can be compiled to wasm without emscripten using wasm32-unknown-unknown.
An example of using cargo-web with the stdweb rust project:
All I'm getting at is that those present set of truths, that that reality hasn't spread everywhere yet, and it's still worth celebrating when it has and when it goes well.
There are plenty of people given plenty of power in leadership that won't bring themselves to understand technology and have yet to retire.
To expand on this, a lot of finance is searching out and understanding, normalizing or exploiting edge cases.
† (only because I'm familiar with it - I'm positive there are better GUIs)
What's missing for this to effectively kill all typical BI tools like PowerBI, Tableau?
a. Filter / Pivot / Aggregate moving clientside means a lot of load off the servers.
b. Performance in wasm is surprisingly close to same C++ code running natively on the desktop.
c. Ability to reuse existing C++ codebase and use it to build both native and web apps.
What is the performance of doing it this way vs. without WASM and workers?
Until we see numbers, this smells like a recruiting play.
> What is the performance of doing it this way vs. without WASM and workers?
@dman says in another thread that the original was in Python and this was 10x faster. If this met their performance goals, it does the job. It may be possible that another way is better, for some definition of better, but who cares if it works as required?
> Until we see numbers, this smells like a recruiting play.
It's presumably OK not to use it if you don't want to.
And what if they think this may help recruiting? Seems like a reasonable trade to me.
We have some light benchmarks we use for regression testing, but definitely need more work in this area
Going with WC is a sound decision since its trivial to wrap it into ember, react, or any other framework-du-jour... especially so since more and more of them are adopting WC patterns and design practices.
I might try to wrap this into react and will see if bosses permit opensourcing it!
I read your docs, and found nothing about how to actually render stuff to the page.
(Sorry about the formatting.... I tried <code>).
Anyway, TL/DR it cannot find boost.
$ ./node_modules/.bin/lerna run start --stream
lerna info version 2.8.0
@jpmorganchase/perspective: > @firstname.lastname@example.org start /home/dj/usr/src/perspective-clone/packages/perspective
@jpmorganchase/perspective: > npm run compile && (npm run perspective & npm run compile_test & npm run compile_node & wait)
@jpmorganchase/perspective: > @email@example.com compile /home/dj/usr/src/perspective-clone/packages/perspective
@jpmorganchase/perspective: > mkdir -p build build/wasm_async build/wasm_sync build/asmjs && (cd build/; emcmake cmake ../; emmake make -j8; cd ..)
@jpmorganchase/perspective: -- Configuring done
@jpmorganchase/perspective: -- Generating done
@jpmorganchase/perspective: -- Build files have been written to: /home/dj/usr/src/perspective-clone/packages/perspective/build
@jpmorganchase/perspective: Scanning dependencies of target psp
@jpmorganchase/perspective: [ 1%] Building CXX object CMakeFiles/psp.dir/src/cpp/base_impl_win.cpp.o
@jpmorganchase/perspective: [ 2%] Building CXX object CMakeFiles/psp.dir/src/cpp/arg_sort.cpp.o
@jpmorganchase/perspective: [ 4%] Building CXX object CMakeFiles/psp.dir/src/cpp/calc_agg_dtype.cpp.o
@jpmorganchase/perspective: [ 5%] Building CXX object CMakeFiles/psp.dir/src/cpp/aggspec.cpp.o
@jpmorganchase/perspective: [ 8%] Building CXX object CMakeFiles/psp.dir/src/cpp/aggregate.cpp.o
@jpmorganchase/perspective: [ 8%] Building CXX object CMakeFiles/psp.dir/src/cpp/base.cpp.o
@jpmorganchase/perspective: [ 9%] Building CXX object CMakeFiles/psp.dir/src/cpp/base_impl_linux.cpp.o
@jpmorganchase/perspective: [ 10%] Building CXX object CMakeFiles/psp.dir/src/cpp/build_filter.cpp.o
@jpmorganchase/perspective: [ 12%] Building CXX object CMakeFiles/psp.dir/src/cpp/column.cpp.o
@jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/cpp/calc_agg_dtype.cpp:11:
@jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/calc_agg_dtype.h:12:
@jpmorganchase/perspective: In file included from /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/schema.h:13:
@jpmorganchase/perspective: /home/dj/usr/src/perspective-clone/packages/perspective/src/include/perspective/base.h:29:10: fatal error: 'boost/unordered_map.hpp' file not found
@jpmorganchase/perspective: #include <boost/unordered_map.hpp>\n ^~~~~~~~~~~~~~~~~~~~~~~~~
I don't know the Cakeshop developers personally, but it looks slick!
Good on whoever took on legal to get this out the door though. Hope you make it to MD or whatever it is you were gunning for.