Hacker News new | past | comments | ask | show | jobs | submit login
10 Years of Open-Source Visualization: Did I learn anything from D3.js? (observablehq.com)
382 points by serial_dev 48 days ago | hide | past | favorite | 52 comments

D3.js is, to me, one of those tools that makes things possible rather than making things easy (although I only use javascript once every year or two so it's always a slog). This is big: it's like a zero-to-one step, rather than incremental progress.

This fall I made an interactive visualization of the fine-grained plate tectonic model I have built for the Himalaya and Tibet for a research project[0]. The model building process (i.e., finding the relative motions of the plates) is largely solving a system of linear equations, so it's not immediately apparent what the results really mean and whether they are acceptably realistic.

D3.js allows me to explore and understand my own scientific results so much more thoroughly than the static quiver plots that I was making in Matplotlib--being able to pan and zoom a globe (not just a projected map!) and see the tectonic plates move through time is phenomenal. Being able to share the results so trivially as a web page is a huge force multiplier as well.

Thanks so much, Mike!

[0]: http://rocksandwater.net/blog/2020/11/tibet-blocks-js/

I can't imagine writing the amount of code Mike has written. The quantity and quality really amazes me. He's all over the internet.

I love D3.js

Here is my favorite example - try click the 4 buttons at the top:


That’s also one of my favorite examples, but for sentimental reasons: it was the first time I met Shan Carter, who came to my talk on D3’s force layout. This eventually led to me working for The New York Times, and then some years later, Shan joining me at Observable.

I love this graph. I had just landed my first programming gig in a statistical startup when this was published and I remember how inspired we were at seeing it. It really opened our eyes to how useful animation can be in data visualization. So much so that we started migrating many of the graphs we had in R to D3.

Here is an example of a graph we did where we blatantly ripped off one of mbostock’s other NYTimes graphs:


That’s an amazing example! I’ve had the pleasure of trying out the mapping part of D3 for a client job recently and while it’s been a bit overwhelming, seeing demonstrations like that really show how powerful it is. And, to some extent, why it’s so overwhelming!

I’d love to see that same breakdown for the $1.9T “covid relief” bill in Congress this week.

The NYT graphics team kills it every time.

Which other libraries has NYT used, or do they have something custom?

I’m impressed also with what I’ve seen, but I don’t know what they do.

D3 is, in fact, their "something custom". Mike Bostock developed it while working at NYT.

I did not know that! That explains a lot!

It is not true. D3 was built in Stanford in Jeff Heer's lab


Following this thread.

Wow, that is an incredible graph. If only more data could be presented so well.

That's an example of misleading graph. The area of a circle makes proportions hard to assess correctly. https://en.wikipedia.org/wiki/Misleading_graph

That link doesn't provide any support for the claim.

I love D3 and am always shocked by Mike Bostock’s willingness to help. Ten years of D3, yes, but ten years also of Mike answering questions in great detail across GitHub, SO, Twitter, etc. And that’s on top of maintenance + running a startup. Some people just have unbelievable drive.

In other news, I’ve enjoyed using D3 over the years, and always jump on visualization projects as a result. Here’s my favorite thing I’ve made while I was learning D3: https://www.jwilber.me/permutationtest/

This is a super cool representation, that made a lot of sense and the animations are super responsive and easy to follow.

Did you open source that page?

Thanks! You can view the page source directly to see the code :)

A little D3 war story: I wrote a solver for the Schrödinger equation, which would render a wavefunction with D3 and SVG - cool right.

It was very slow to render. Eventually I found out why: D3 was round-tripping the SVG through text! The render time was dominated by the browser parsing the thousands of 15+ digit floating point numbers.

I added a call to round the floats to two decimal places, so they would be textually shorter, and it was a huge speedup. What a strange optimization.

I later rewrote it using WebGL; here it is should you care: https://ridiculousfish.com/wavefiz/

I’ve written some animations in React and Svelte recently which edit SVG properties to move objects, and I found myself making the same rounding optimisation (along with using a single path to draw many circles, rather than many circle objects). I think it’s pretty wild that the only way to position objects in the DOM is with strings.

I found this so useful. I'm building a suite of open source tools at https://datasette.io/ and the section "Teaching is the most impactful aspect of tool building" was something of a light bulb for me.

"To maximize your impact, then, teaching must be central to your strategy. This means documentation, tutorials, examples, videos, tweets, and more."

I have pretty comprehensive documentation ( https://docs.datasette.io/ ) but almost nothing in the way of tutorials - and I added the first demo video to the website just last week.

This has given me a ton to think about. Thanks, Mike!

FWIW, I accumulated a bunch of docs writing resources while rewriting the Redux core docs tutorials last year:


I specifically recommend Divio's "Documentation System" structure, and the Vue docs "Writing Style Guide" section as being particularly good resources:



And here's one of the new tutorials I ended up writing, which took inspiration from many different sources:


> Of all forms of documentation, examples seem to be the most effective.

This is applicable in almost all types software, not just graphical ones like D3. Many software lack this and can be made much better/attract more adoption if there are more examples.

For all the hate people have for PHP, I must say their documentation is extremely easy to consume.

Every function/method/class has code examples of actual usage (and many times multiple examples for different use cases).

It also provides a section where people can post documentation comments per function/method/class.

This has been there since nearly day 1. Wouldnt surprise me if this helped considerably in its overall adoption as a language.

I honestly wish more language documentation provided this kind of format and accessibility.

I love this post. I can't tell you how many times I've started trying to use d3 and gave up after a couple hours because "all I wanted was a bar graph" (at least 3). Learning it is still on my bucket list because it is truly an amazing library.

These days I’d recommend sticking to a framework you already know (like React or Svelte) for DOM manipulation and using D3 as a library on top of that. It eliminates a lot of the learning curve that was associated with D3’s elegant but esoteric approach to DOM manipulation. D3 is still an excellent way to do scales, force-directed graphs, grids and ticks, etc. and it’s modular enough that it works well whether it owns the DOM nodes or not.

(Shameless plug: during a two week quarantine isolation I started but did not finish making a video series on using d3 with Svelte: https://youtube.com/playlist?list=PLXiRqCwhpnehz5z0gpsmcb2l8...)

Last time I looked into this, React and D3 didn't really work together, since D3 manipulates the DOM and React has its own Shadow DOM. Has this changed?

D3 still manipulates the DOM if you ask it to, but only a few of its subpackages actually manipulate the DOM directly (mostly the selection, axes, dragging, and zooming APIs IIRC). Everything else either transforms data structures within JavaScript, or generates SVG path strings which you can pass down into path elements through JSX.

The one place I’ve found it doesn’t quite play nice with React is not the DOM at all, but that the force simulation insists on modifying data in place where React wants things to be immutable, but it’s not a dealbreaker.

This is an excellent advice. And as you do more advanced graphing you learn more about SVG, which opens even more doors for even more advanced or customized graphing.

With SVG as your canvas and a good framework for DOM manipulation (in my case it is mostly vue), and the shapes and scales available from D3, it really starts to feel like you have as much freedom as you would with a pen and paper.

Could anyone recommend a good book on the subject(s)?

Check out Amelia Wattenberger’s work. She’s done a lot of writing and has created a course and written a book. I can’t speak directly to the book but I think her work in general is a good representation of the modern way that D3 is used in conjunction with other front-end frameworks.


Second this, reading her work is how I was able to build some really cool things using those ideas. Once you get it, it really unlocks the potential to create some unique things

Her d3.js book is awesome. Learnt a lot from it.

Most of the the fullstack (renamed to newline) books are pretty good.

No affiliation just a happy customer.

This is so hard to mesh existing examples into your framework of choice though. That could do well with a lot more examples around to spot the line between framework, DOM and D3

You're looking for https://c3js.org/ :)

My go-to for that has been Vega these days. Just love the conceptual model and I've actually been able to hold the key parta in my head, compared to most such tools.

For standard graphs, there are plenty of simple libraries built on top of d3 or just pure javascript. Use those. D3 is for when you want to create something custom or complex.

I think this is great:

> The only way to recognize — and then bridge — the gulf of understanding between you and people using your tool is to observe their struggles and talk to them. It’s amazing how quickly glaring flaws are revealed this way. Answering questions on Stack Overflow (or GitHub, Twitter, Slack, or wherever else) is not a selfless act of altruism; it’s a chance to learn, to find where people struggle and hear their perspective. Each question is an opportunity to help one individual, but that question can also inspire a tutorial, an example, or even a feature that prevents others from hitting the same issue.

What I love about being a front-end engineer is seeing how people use what I build, which almost always brings up "obvious" issues and improvements that weren't so obvious when you're working on it yourself.

Now that I'm mostly building developer tooling, this gets even better: not only are the users like me so I can empathise with them better, but I know where they're going with their questions and observe them "in the wild", and easily figure out what help they need without needing to set up specific UX research sessions.

But this primarily gets fun if you use it as a way to scale up your support. Which means it's important to keep the following in mind:

> But there’s also a limit. As one person, you can’t help everyone. And you can’t make everyone happy because your tool is only a tiny part of their life. Don’t make that your goal: focus on learning and broadening your perspective. If it stops being constructive for you, then stop. Period. And don’t feel bad!

Don't know a damn thing about visualization or how to use d3.js but I have a shitload of respect for Bostock's work on it, and his role in using it at the NYT.

Love this window into his thought process.

I'm also happy to see someone as accomplished as him say that randos on the internet are also capable of hurting his feelings. It makes me feel substantially less alone in that regard.

I owe a lot of my career to people like Mike Bostock who have made open-source libraries like d3 free and available to the world. I'm sure I'm just one of countless developers who feel the same like me. Eternally grateful

In some ways I miss my time as an R programmer. Back then I could type R and make various sorts of rough useful plots of data quickly. 10 years of nonstop programming later, I no longer use R, and I cannot make rough useful plots quickly in either Python or JS. (There is nothing quick about activating the right virtualenv, starting a jupyter server, getting a notebook running, and trying to remember how to import and use matplotlib. And JS is worse).

Have you tried the python port of ggplot, plotnine? I screencast doing live data analyses with jupyter notebooks, virtualenv, and plotnine. I'm definitely much quicker in R, but it's not too bad!


This might be slightly off-topic, but I can't seem to find a direct solution.

I'm looking to build a complex dependency/timeline graph for a web of interconnected stories in a media franchise. Each story has a release date where it needs to appear, and an arbitrary list of dependencies for previous episodes/stories/interludes which would have a line drawn to that dependency. The end result would be a sort of hybrid digraph/timeline, enabling a reader to get the 'minimum path' to catch up to a certain story in the collection.

D3 looked promising, but there's no direct equivalent I can see in the examples. For those that are experienced with D3, is that sort of thing achievable? Should I be looking at D3 or something else?

MIT's Exhibit timeline widget used to be really nice for this, or at least extremely customizable timelines, but I am not sure of the state of that project... http://www.simile-widgets.org/timeline/

this should be feasible with D3. The way I understand it, is that you are trying to make node graph along a time series. So you basically just have one axis (date; usually plotted on the x-axis).

The way I would plot it is to have a long horizontal scroll along the x-axis where a series of nodes are scattered in the single date dimension. Hovering (or clicking) over this node would expand a list of dependencies into the second dimension (distance from the original node might represent time; perhaps in two columns, past [depends on] and future [depended by]). Clicking on one of these dependency nodes would scroll you smoothly where this dependency nodes reside on the main axis.

All of this is possible (and almost trivial) with D3. However you might need to be innovative in implementation as I don’t think there are easy examples for you to follow.

> To maximize your impact, then, teaching must be central to your strategy. This means documentation, tutorials, examples, videos, tweets, and more.

I think this is one of the main reasons the FastAPI framework is very successful. The documentation teaches you about everything and doesn't assume any prerequisites besides general Python knowledge. I've used it recently, and it's such a delight to work with.

I used d3.js for one of my first programming projects. I was grateful and impressed that almost every question on Stack Overflow was answered by Mike Bostock or Jason Davies.

I'm (a little) older and (a little) wiser, and know this kind of support is a huge amount of work. It sounds like "focus[ing] on learning and broadening your perspective" is what fueled all those answers I so appreciated.

D3.js is cool - and you can do more with it than just data visualisation https://datacrayon.com/posts/visualisation/visualisation-wit...

I like D3 and Observable! So well done experience for the users!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact