Day 9 - D3 for SysAdmins

Written by: Anthony Elizondo (@complex)
Edited by: Shaun Mouton (@sdmouton)

In this post we will talk an introductory look at D3.js and using it on relatively "raw" datasets such as those in CSV or JSON.

D3.js is a powerful Javascript library that allows you to build SVG images and animations, implemented with HTML5 and CSS.

There are some who will tell you D3.js has a steep learning curve. I think the reason for this is it lives at the center of a Venn diagram where the circles are Javascript, CSS, math, and graphic design. Not the most common skill set.

But don’t let that scare you. You’re awesome! You can do it!

Visualizing your Data

Are you collecting metrics? A recent survey by James Turnbull of Kickstarter says that 90% of you are. (If not, why not?) Ideally these metrics are fed into a permanent installation of a tool that can be used for measuring performance, alarming on errors, or just general trending. And that is great.

But perhaps you just want to whip up something on an ad-hoc basis. D3.js is superb at this. With simply a CSV file and a web server (even something as simple as "python -m SimpleHTTPServer 8000", see William Bowers' list) you can create something that works on any modern browser, mobile included.

Data Type and Format

Before we can do anything, we have to think about what type of data we have. More precisely, what is the story we want to tell about it? We might have a collection of ohai output (JSON) from 10,000 servers. This type of data would be nicely visualized with a force-directed graph, perhaps even sorted by type. Maybe you have logs of concurrent sessions on HA Proxy or your F5 load balancer. Then this heat map would work well. Your data might be as simple as a single metric, but you want to chart it over time. A simple line chart would be sufficient.

Fundamentals of D3.js

D3.js operates on the DOM of a web page. You can boil most of its operation in three phases. First, it creates SVGs and adds ("appends") them to the page. Next it reads in data embedded in the page itself, from a separate file, or from other online source via AJAX. Then, it performs transforms to the SVG elements based on this data, or perhaps based on user input.

Simple Example

Now we’ll try create a simple example. Assume our data, after some awk mangling, is in a TSV. The first column is the date in YYYY-MM-DD format, the second is a scalar indicating how many servers we have.

date   servers
2013-12-08  4343
2013-12-07  4328
2013-12-06  4325

First, include D3 in our HTML file,

<script src="http://d3js.org/d3.vs.js"></script>

Now create the SVG.

var svg = d3.select("body").append("svg")
.attr("width", 860)
.attr("height", 500)
.append("g")
.attr("transform", "translate(" + margin.left + "," + margin.top + ")");

Time to load our data.

d3.tsv("vmcount.tsv", function(error, data) {
data.forEach(function(d) {
d.date = parseDate(d.date);
d.servers = +d.servers;
});

And draw it.

svg.append("path")
.datum(data)
.attr("class", "line")
.attr("d", line);

The .append("g") and attr("d", line) are not magic. They are SVG element tags. The "g" attribute indicates "group everything together as one". The "d" attribute defines a path to follow. The 3 bits of Javascript above are the core of the work, but they alone are insufficient. There is a bit more required to bring it to life and make it look nice. That includes defining axes, scales, labels and domains (in the geometric sense). The full working example can be found here. It adds all the features mentioned above, plus a fancy hover to show precise values.

On Your Way

Hopefully I’ve demystified D3.js for you a bit with this short introduction. To dive deeper I suggest browsing some of Mike Bostock’s simpler examples, and don’t be afraid to ask your friendly neighborhood frontend developer for help! :) The full D3.js API can be found here.

Resources:

• Interactive Data Visualization for the Web by Scott Murray (@alignedleft).
• D3.js Data Visualization Fundamentals by Ben Sullins (@bensullins).