
Show HN: Rendering One Million Datapoints with D3 and WebGL - ColinEberhardt
https://blog.scottlogic.com/2020/05/01/rendering-one-million-points-with-d3.html
======
zylepe
Very cool! I’ve done visualizations on similar size datasets by mapping tsne
output to fake latitude/longitude coordinates and rendering it with mapbox gl.
It feels like a hack, but then you get all of the panning/zooming interaction,
styling, and rendering labels with collision avoidance for free. For smaller
datasets I pass in geojson directly, but for larger ones I precompute vector
tiles so you can lazy-load more of the data when you zoom in. There are a lot
of things built for mapping that are relevant for other domains, I wish that
there was a common base that could be extracted that mapping libraries adapt
to geographic data but data visualizations could adapt to non-geographic data.

I’ll have to play around with your technique more, encouraging to see you get
good performance eagerly loading a million data points!

~~~
7373737373
I can recommend LargeVis[0][1] which I used to layout and Polymaps[2] to
visualize tiles of the [https://wikiscape.org](https://wikiscape.org) article
point cloud.

[0]
[https://github.com/lferry007/LargeVis](https://github.com/lferry007/LargeVis)
[1] [https://github.com/elbamos/largeVis](https://github.com/elbamos/largeVis)
[2] [http://polymaps.org/](http://polymaps.org/)

~~~
7373737373
Scrap polymaps, use leaflet instead:
[https://leafletjs.com](https://leafletjs.com)

------
1wheel
Nice write up Colin!

fwiw quadtrees don't make much sense when you're showing a tooltip for ~1
million points: [https://roadtolarissa.com/scan-
sorted/](https://roadtolarissa.com/scan-sorted/)

~~~
ColinEberhardt
That's really interesting, thanks for sharing those results. So easy to go
down the route of premature optimisation!

------
paulgb
Scott Logic is probably the highest-quality company technology blog I've
encountered in a while. I keep returning to it for WebAssembly and WebGL
topics and it is consistently high quality. I'm not even sure what Scott Logic
does, but they sure know their stuff :)

Kudos, Colin and team.

~~~
ColinEberhardt
Thanks Paul, really appreciate that :-)

Just FYI, we're a medium sized software consultancy based in the UK. We do
quite a lot of work in financial services, which is why you'll find quite a
few posts on performance and cutting-edge web technology. I started the
company blog 12 years ago and now we have ~50 authors creating some amazing
content. Unfortunately I don't have the time to write quite as much as I used
to!

------
gdubs
The scatterplot of the library data is fascinating — looks like an organism.

