
Whatever It Takes: Visualizations in E-mail with d3.js - pixelmonkey
http://blog.parsely.com/post/34241210620/whatever-it-takes
======
Colliwinks
I am amazed that their 'huge revelation' was that perhaps they shouldn't be
doing huge amounts of dynamic image generation client side.

Is the point of this article "we managed to get charts into email"? In places
there is a bit too much subjective self-praise for anything other than a sales
pitch: "...elegant, beautiful, and timely..." I personally think their
greyscale and doughnut charts don't look great.

~~~
gruseom
Please don't post comments like this. It adds no value and it violates the HN
guidelines, since no one who isn't a jerk would say this to someone's face
when they were showing their work. You would simply keep it to yourself.

~~~
debacle
"no one who isn't a jerk would say this to someone's face when they were
showing their work."

That's just your opinion, and in my opinion you are being far more offensive
than Colliwinks by calling him a jerk on a public forum.

~~~
gruseom
On the contrary, my assumption is that Colliwinks is _not_ a jerk and would
never have said that in person, and therefore broke the guidelines (presumably
unintentionally) by saying it here. That's why I said "to someone's face" -
it's the most important part.

------
svnv
"I once attended a one-day course by the renowned Information Visualization
educator, Edward Tufte."

And still they used doughnut charts.

~~~
bcoates
Serious question, what's wrong with doughnut charts?

~~~
mjohn
The same reason that pie charts are not very effective - it's relatively hard
to make comparisons between angles instead of lengths like in a bar chart.
It's probably even harder with a doughnut.

Here is a PDF article by Stephen Few about pie charts:
<http://www.perceptualedge.com/articles/08-21-07.pdf>

~~~
bcoates
That's a compelling article and I think I'm going to be a lot more cautious
about using pie charts in the future, but the thin-doughnut design they're
using here looks OK to me.

When you're summarizing coarse relative size across arbitrary categories, bar
charts and tables give your selection of categories too much weight: everyone
looks at the biggest bar or the biggest number and it's really hard to measure
a long tail distribution visually.

If the message you're trying to send is, "nothing but X and Y matter,
everything else combined is tiny", I still can't imagine a better
visualization than a pie (or a doughnut).

~~~
codementum
You're absolutely right. Everyone harps on pie charts for being awful for
element-to-element comparisons. This mainly stems from Tufte's work and
experiments from Cleveland and McGill.

However, in part-to-whole relationship tasks, pie charts can outperform bar
charts, since in bar charts there is no true "whole".

(In addition to Few's post, Kosara has more on this:
<http://eagereyes.org/criticism/in-defense-of-pie-charts>)

~~~
cpeterso
A composite bar chart is like a pie chart that has been unrolled. The length
of the bar is the "whole".

[http://www.ncsu.edu/labwrite/res/gh/gh-
bargraph.html#composi...](http://www.ncsu.edu/labwrite/res/gh/gh-
bargraph.html#compositebar)

------
bdunn
Beautiful. I ended up resorting to RMagick and some custom charting code to
build personalized infographics for the weekly digests that Planscope sends
out.

<https://dl.dropbox.com/u/2205912/weekly-digest-planscope.png>

Weekly digests + sexy charts = unbelievable upticks in retention. It's also a
great way to push product updates and blog posts directly to your customers.

~~~
pixelmonkey
That looks awesome and engaging. Great work.

------
kungfoo
Awesome article! Loved the mashup in technologies.

Just wanted to point out: Although the title/article emphasizes d3, I think
additional emphasis should be put on PhantomJS. There are a lot of
complexities with implementing a headless browser. Libraries, such as Rhino
and Selenium, have fallen short. PhantomJS really has created a nice balance
for developers.

~~~
unreal37
Both D3 and PhantomJS look very interesting. Server side rendering of
JavaScript... Some cool stuff still to be made with that.

------
goodside
Parse.ly: Please make the header on your blog link to parse.ly, not
blog.parse.ly.

~~~
pixelmonkey
Great idea. Done.

------
harshaw
Bottom line: if you need reports you should be thinking about phantomJS. There
are other projects like wkhmltopdf but I think phantom is better and more
flexible.

Phantom sites on top of QTWebKit (and more specifically QTWebFrame/Page) so
the code is quite approachable if you want to look under the hood. I think the
only downside is that QTWebKit lags a bit so a competing technology built on
chromium would be even better. The other downside of course is that your
rendering needs are significantly higher than generating text based email (if
you are at significant volume).

~~~
harshaw
I should also note: if you want to go this approach you might have luck
embedding the generate report image in a dataURL so the user doesn't have to
turn on images to see the data. However, I don't know if that works in all
browsers or web mail clients.

------
rforte
Great article. We are doing something very similar with phantomjs to generate
daily digests of our client dashboard for customers. Are you pointing the img
tag to an external URL or are you inline embedding the image in the email? I
notice that some email clients like mail.app and sparrow will replicate the
attached image twice when you inline embed the image as an attachement. This
is because it will render the body of the email but then show the attachement
below the body (as an attachement preview). Wondering if you guys ran into a
similar issue.

------
don_draper
Could someone point out examples of visualizations that make a big difference.
I've spent quite a bit of time creating maps and graphs for corporate bigwigs
and it always just seems like eye candy. It seems like a list with
percentages, instead of fancy graphs, is often just as good and way cheaper.

Aside: I understand trends like stock value over time can be useful to see in
a chart, but often times other charts seem unnecessary to me.

~~~
sebg
<http://en.wikipedia.org/wiki/Anscombe%27s_quartet>

Four Data Sets:

Simple Statistical Summary of Each Data Set is Identical.

Visually Totally Different.

edit: fixed url - thanks @eddmc

~~~
eddmc
Your link should be: <http://en.wikipedia.org/wiki/Anscombe%27s_quartet>

Interesting stuff!

------
gyepi

      I wondered: could we perhaps get PhantomJS to run our data
      visualization on the server, capture the rendered output, 
      and then assemble an email with light HTML, CSS, and 
      rendered images?
    

This server side rendering technique is a quite powerful one. I've used it in
a couple of projects and expect to see it used more often as people
incorporate it into their tool kits.

~~~
Evbn
Much more brittle, handicapping, and more expensive to boot. What if the user
wants to zoom in a portion of the data, ignore all but top 3 categories, pan
over 100s of points in a series?

~~~
jeromeparadis
Well, since were talking about a technique to avoid email rendering
limitations, just add some clicks to the Web UI to allow digging in further.

------
allertonm
This is similar to how we do our email delivery of dashboards at Indicee
(<http://www.indicee.com>)

We don't use D3 for our visualizations (I'd love to, but our users can use the
dashboards "live" and, well... Internet Explorer 8) and we snapshot the entire
dashboard - which isn't quite as clever as what is done here, but does have
the advantage that the resulting mail is self-contained and does not require
the dynamically generated images to be hosted somewhere.

Like these guys, I have nothing but good things to say about PhantomJS. We'd
previously used another "headless webkit" - wkhtmltoimage - in a similar role
but Phantom's design as a general purpose scripting engine for headless
browsing tasks is much, much more useful.

------
kimmel
I know I cannot be the only person who sees this and thinks text only emails
are still the way to go. If you are going to generate something like this
maybe you should just make it a pdf and attach it to the email. HTML emails
are still a bad idea.

~~~
bkanber
Why do you think HTML emails are a bad idea?

------
krrrh
Interesting approach, and nice work. I did something similar a few years ago
to generate dashboard screens for a digital signage product. I used svg
templates I'd initially create in Inkscape, and erubis to handle XML valid
embedded ruby. Inkscape has an amazing command line that handled converting
the resulting SVG document into PNG or PDF.

I liked the idea of using regular HTML and CSS, but pre-phantomjs the closest
I could come was princeXML, and even then it lacked js support and the
licensing costs were pretty high.

------
atesti
I looked through the parse.ly homepage and even in the source code of the blog
post, can someone answer a quick question:

How does parse.ly actually work? Do you sign up for Google Analytics and open
it to them to they get the data from there? Or do you embed a piece of
javascript from parse.ly which tracks the referers, etc. in their database?

I did not want to sign up just to learn this and there was no technical
information or FAQ on the homepage or I overlooked it!

Thanks!

~~~
pixelmonkey
You embed our JavaScript tracker on your site.

It's integrated similarly to Google Analytics. See
<http://parse.ly/api/tracker.html> for more technical information on our
JavaScript tracker.

Sorry that you couldn't find that information easily -- I'll try to think
about how to make it clearer on the website.

------
thefreshteapot
Wonderful article!

Could you tell us a little more about domshot.

I just the other day looked at phantomjs and ended up using casperjs on top as
it allowed to use standard selectors to pick the elements I wanted to capture.

~~~
pixelmonkey
domshot is a Python API that lets you include JavaScript / CSS modules (e.g.
jQuery, d3.js, your brand style). You then render PNGs using from Jinja2
templates. Under the hood, what it does is render static HTML from the Jinja2
template, open a PhantomJS headless browser instance on that page, and then
render it into a PNG.

This way with a few lines of code, you can go from Jinja2 template -> rendered
PNG without having to worry about the details of passing information back-and-
forth between your Python app and a PhantomJS instance.

Sound interesting? We're planning on open sourcing it. Ping us (hello at
parsely dot com) and we'll notify you of when it's available.

It would be open source already, but we're a bit backlogged on spare time
projects at the moment. We just made an open source release of
<http://schema.to> a couple weeks ago, and now Sandy struck so we find
ourselves catching up. But, soon!

------
tiedemann
This made me remember the awesome feature CreateHtmlBody (in CDOSYS) for
classic ASP. Many mails were sent using that :)

------
nachteilig
Nice that d3.js is getting so much press these days. Really great little
library.

------
pdog
Is the text searchable and selectable?

~~~
pixelmonkey
Yes. With the exception of labels within the visualizations themselves.

Here is a "before" screenshot of the Weekly Digest without the images
downloaded in GMail:

[http://farm9.staticflickr.com/8333/8148522629_b92e702a91_o.p...](http://farm9.staticflickr.com/8333/8148522629_b92e702a91_o.png)

And here's an "after" screenshot, after the images have been downloaded and
rendered into the mail as well:

[http://farm9.staticflickr.com/8326/8119725381_e05bf9f983_o.p...](http://farm9.staticflickr.com/8326/8119725381_e05bf9f983_o.png)

