Day 10 - Assembling Your Year In Review

By: Paige Bernier (@alpacatron3000)
Edited by: Jennifer Davis (@sigje) and Scott Murphy (@ovsage)

Intro

There are a few moments in my career that I have been struck by a story told with data. When I set out as a Site Reliability Engineer into the big wide world I wanted to capture that data storytelling magic and have adapted a presentation I call the “Year in Review”.

My first company had a tradition of taking a moment to pause and review the year by the numbers. The showstopper was the chart showing the amount of data ingested year over year since the founding.

In a single glance that chart conveyed a story that would take hours to tell!

It communicated the incredible efforts the employees took to scale the system to handle ingesting, processing, publishing and storing an ever increasing mountain of data. It illustrated how far the company had come and we were confronted head on with the realization that “what got you here, won’t get you there”.

The biggest impact I have seen comes after the presentation. Discussions from Year in Reviews have sparked sweeping oncall management changes as well as minor, but important, changes in the way developers engage with the SRE team.

Before diving into implementation details, let’s look at why this type of data storytelling is such a powerful tool by examining the core purpose of SRE

The Mission of SRE

The mission of an SRE team is to improve system reliability by facilitating change.

System reliability is the sum of hundreds of decisions humans make when developing, deploying, and maintaining software systems; it is not an intrinsic property¹ of the systems (Patrick O’Connor, 1998). SRE job descriptions tout phrases like “evangelize a DevOps culture” and “influence without authority” acknowledging our roles as change agents.

And as often heard, “change is hard”. As change agents, we are often faced with conflicting priorities, multiple stakeholders internal and external, and fear of the new and unknown.

However, just as often we hear “change is the only constant”. Whether it’s hardware improvements, operating system upgrades, security vulnerability announcements, software dependencies, or the software that we manage as a service, we are constantly monitoring and implementing change.

Combine these two axioms, for extra difficulty:

Ask any engineer who has been forced into a major operating system upgrade when the version of software they’re running requires the previous OS.

As an SRE I often want to make changes across the entire engineering organization such as developing oncall onboarding, ensuring that we are monitoring the customer’s experience, clarifying the lines of responsibility between developers and operators and more!

These types of changes that affect everyone is difficult to effectively implement until two things are true:

• Is there a shared understanding of the current state?
• Is there agreement that the current state needs to change?

This does not mean there needs to be consensus on what changes need to be made!

Is there a shared understanding of the current state?

The answer to this can be a resounding “Yes!” after your Year in Review presentation. Here’s why:

Humans learn best from stories, feelings, senses, and opinions commonly known as qualitative data. Focusing only on these exclusively you risk coming to broad conclusions without nuance or context.

Businesses claim to operate on data, facts and figures, or quantitative data. Focusing purely on the numbers you risk having too many details leading to irrelevant rabbit holes.

In fact, the two seemingly disparate viewpoints aren’t at odds at all. You can even validate findings by using the other category of data.

Feel: “Our monitoring sucks, none of the last 5 pages I got were actionable”

Fact: The primary oncall was paged 5 times out of business hours last week

Finding: Team X is getting paged frequently for non-actionable reasons

Hosting a “Year in Review” means weaving a story using the quantitative data about what occurred in your systems with the qualitative “anec-data” from a human perspective to build a foundation to introduce change.

Is there agreement that the current state needs to change?

This is a more complex endeavor - identifying and implementing change is the hard work of collaborating across teams, roles and competing incentives, motives, and needs. Think of “Year in Review” as a springboard for driving discussion and debate to align on “do we agree something needs to change?”

What does this look like in practice?

At a previous company I heard from engineers and managers alike that the oncall rotations were in need of a shake up. This was an excellent starting place where everyone agreed that there was a problem but was having trouble implementing the necessary changes.

With a goal in mind to identify what exactly the oncall issues were my team tailored a “Year in Review” focused mainly on oncall metrics such as alert noise, hours oncall per engineer, pages received per engineer. Slides illustrated the deluge of alert storms no human could possibly investigate in a given shift and were largely unactionable noise. The impact of not addressing this problem was clear, we were likely missing important signals in the noise and oncalls weren’t able to effectively prioritize their time.

After reviewing the data as a group, my team facilitated a brainstorm to address the barriers to changing the rotations:

• How to handle ownership when multiple teams contribute code?
• What are the “hot potato” services no one feels comfortable owning?
• What services are unofficially owned by a single engineer that needs documentation?
• What is the goal of a low urgency or warning alert?

Based on the main discussion and others in standups and sidebars, my team proposed new team-service ownership and rotations. Several weeks and few rounds of revisions later we merged the PR with our new Terraformed oncall rotations!

DIY “Year in Review”

So, how do you create a “Year in Review” for an SRE team? To start, I typically have a few things in mind about what I think happened and what the data will show. It is fascinating to see where your perception of the system and reality diverge. You can kick off your process by asking a couple of questions:

• What story are you expecting the data to tell?
• What changes do you think need to be made in the next year to improve reliability?

1. Book a meeting with all parties (including engineers, managers, sre, qa, ops, product managers). If there is an existing meeting like an All-Hands or Demo Hour sign up for a presentation slot
2. Kick off a brainstorming session and have participants list out possible changes to include. Such as new features launched or infrastructure expansions to new regions, or even doubling the organization size.
3. Ask teams (including managers)
   1. What data they would find interesting
   2. What data they could contribute from their domain
4. List the company-specific tooling for data sources like:
   1. Version Control
   2. CI/CD
   3. Monitoring
   4. Incident Management
   5. Ticket tracking system
   6. Documentation store
   7. Support ticket system
5. Enlist the help of others to gather the interesting metrics over the past year or year over year. Some suggestions are:
   1. Noisiest alerts
   2. Number of environments
   3. Oncall engineers
   4. Number of services
   5. Ratio of oncall engineer to number of services oncall for
   6. Age of dependencies/libraries
   7. # of hours oncall per person
   8. Number of features launched
   9. # of after hour pages
   10. Ratio of warning alerts to pages
   11. Number of production deploys rolled up by day
   12. Number of open incident AIs
   13. Ingress traffic or other indicator of system load
   14. Most viewed documentation pages
   15. Most search documentation terms
   16. Time to first PR
   17. ….and so much more!
6. Slice and dice the data trying out top 10 lists, total sum, or segment by using whatever constructs your company has such as:
   1. Department
   2. Service
   3. Team
   4. Product Feature
7. Group the data into themed areas “oncall” “production” “onboarding” etc. If you have convinced folks to co-present with you each person can be responsible for presenting a different theme
8. Assemble into a slide deck with one chart per slide to maximize impact
9. Hold the meeting and present your findings,
10. Discuss! In the meeting, after the meeting before the next Year In Review how you interpreted the data compared to others
11. Publish the data and your queries so everyone can explore and answer their own questions

Parting Thoughts

SREs are uniquely suited to facilitate a Year in Review bringing a system-wide perspective on the people, processes, and technology and mission to improve reliability. Keep in mind that much like effecting change, hosting a Year in Review is not a solo effort!

Going solo means you will only capture YOUR thoughts which will almost certainly be tempered by the unique vantage points from others. The more perspectives you invite, the fuller the story of your system will be.

Please share your favorite data storytelling moments or Year in Review stats with me on Twitter at @alpacatron3000

Citation

O’Connor, P. (1998) Standards in reliability and safety engineering [Article]. Elsevier Science Limited, 9 Dec. 2021.

https://www.sciencedirect.com/science/article/abs/pii/S095183209883010X

Notes

---

1. Since the SRE field is still getting established outside of Google, I started to read perspectives from Reliability Engineering in other disciplines. A nugget from Patrick O’Connor’s “Standards in reliability and safety engineering” paper sparked a spicy but important revelation about reliability.

   “Those reliability standards which apply mathematical/ quantitative methods are also based on the inappropriate application of “scientific” thinking. An engineered system or a component has no intrinsic property of reliability, expressible for example as a failure rate. Truly scientifically based properties of systems and components include mass, power output, etc., and these can therefore be predicted and measured with credibility. However, whether a missile or a microcircuit fails depends upon the quality of the design, production, m~nten~ce and use applied to it. These are human contributions, not “scientific”. “ ↩

Day 21 - collectd

Collectd is a statistics collection tool I've recently found quite useful. Other tools in this space include Munin, Cacti, and Ganglia. For starters, collectd can collect at fairly high frequencies (the default is every 10 seconds) relative to other collection tools. The config syntax is consistent and looks similar to Apache httpd's config syntax. As a bonus, it comes with almost a hundred plugins to help you get statistics from devices and applications and into RRD files (and other outputs, if you want). Each plugin has a host of different configuration options it can support, including changing the collection interval.

Reading over the project's website, I noticed a few things that struck me as good things. It's important to mention that the reason I noticed these things was because the project has quite good documentation.

First, there is the the network plugin., which allows you to send and receive collectd data to and from other collectd instances, or anything that speaks the collectd network protocol. Many networking scenarios are supported: unicast, multicast, and even proxying. Second, many plugins have reasonable configuration defaults. For example, the interface plugin defaults to capturing stats on all interfaces. Addtionally, the default plugin set includes ones to capture data from other systems like Ganglia and JMX. Other base plugins allow you to easily fetch values from databases (DBI), web servers (cURL), and scripts (Exec).

The DBI and cURL plugins cover a pretty wide area of uses, and the Exec plugin is useful when you can't find a plugin that does exactly what you want. The benefit here is that you may not have to write a complex script just to fetch a value from a database or webserver just to store that data with collectd. The DBI plugin even supports using specific column values as fields and others as collectd values, rather than having statically defined fields. I like.

Other nice features include the ability to filter collected data, possibly modifying it before it gets written to disk. The project also comes with a useful tool called collectd-nagios which allows you to use collectd data for nagios checks. This lets you make collectd do the hard work of collecting the data and lets you use the nagios plugin to simply set alert thresholds.

When playing with collectd and when reading the docs, I haven't seen any points where I have found myself worrying about the difficulty in automating collectd's configuration.

So, what's bad? Collectd itself doesn't do graphs for you; it acts as a data collection system only. If you want graphs from the RRDs it stores, you'll need to use the decent-but-not-superb web interface called 'collection3' that comes with collectd in the contrib directory. There are other projects, like Visage, that are working on providing a better interface to the data collectd records. I started with collection3, which looks like this:

I circled the navigation overlay that collection3 puts on the graphs. These allow you to pan/zoom around to various time views - a very important feature of any graph view system. Definitely a win.

For fun, I decided to use the cURL plugin to fetch a random number from random.org. That configuration looks like this:

<Plugin curl>
  <Page "random_number">
    URL "http://www.random.org/integers/?num=1&min=1&max=100&col=1&base=10&format=plain&rnd=new"
    <Match>
      Regex "([0-9]+)"
      DSType "GaugeAverage"
      Type "percent"
      Instance "randomvalue"
    </Match>
  </Page>
</Plugin>

Resulting graph looks like this after letting collectd run for a few minutes:

The good documentation and nice set of features were what convinced me to try collectd. The default configuration includes a few plugins and writes out RRDs as expected, and it was easy to add new collections from different plugins (like the random.org curl one, above).

Further reading:

• RRDtool - data storage and graphing system for time series data. Used by collectd, Munin, Ganglia, Cacti, etc...
• cURL is a tool and library for fetching URLs
• things related to collectd on collectd's wiki

Day 20 - Ganglia

Ganglia is a monitoring tool designed originally to help scalably monitor computing grids and clusters. How can it help you, even if you don't run traditional computing grids or clusters?

Ganglia is an RRDtool-based (like Cacti) monitoring and graphing system. Ganglia differs from Cacti in that configuration is much more automatic. Ganglia's design centers around two programs: gmond and gmetad. The gmond program listens for metric reports from other gmond programs (or tools that emit the same messages). The gmetad program periodically polls a single gmond for data on an entire cluster. The trick, here, is that every time gmond gets data, it sends that data via multicast to other gmonds, so every gmond has state for the whole cluster. I presume that the actual gmond used by gmetad is chosen at random, and if the chosen gmond host fails, another gmond host is chosen.

In addition to clusters (one gmetad for N gmonds), Ganglia supports a higher level collection they call a grid. A grid is automatically learned when you have one gmetad polling from another gmetad. I am unaware if you can have more than these levels (host, cluster, grid).

Multicast: This means your network gear will need multicast routing enabled if you hope to span broadcast domains with this monitoring. Alternately, gmond can be configured to send updates to a unicast address which can avoid needing multicast routing and other potentially difficult network features.

Both gmond and gmetad have reasonably easy-to-use configuration files and come with very reasonable default values. Simply running gmond and gmetad from the default configurations will result in data you can access easily. The primary Ganglia human interface is through a webserver.

Getting data out of Ganglia is easy. The historical data is stored in RRD files in a known location organized by cluster and hostname, so you can use your favorite rrdtool interface to query data. The current data is stored on any gmond which is queryable by connecting to ganglia's xml port (default 8649). The service listening on that port will dump the metric data by cluster and host in XML. XML might make you groan, but it's use will help you write tools (like nagios checks) to use the current data.

My first question after playing with Ganglia for a few minutes was, "How do I monitor my network gear?" Typical network gear won't allow you to run arbitrary binaries on them. Luckily, Ganglia comes with a tool for broadcasting metric messages, gmetric. With gmetric, you can spoof the source of a piece of data and easily claim that it came from your switch or router. This tool is also the easiest way to extend the metrics ganglia monitors for you. For example:

% gmetric -S "192.168.0.254:myrouter" -n uptime -v 3644 -t uint32 -u seconds
  spoofName: myrouter    spoofIP: 192.168.0.254 

And the metric reported on the xml port is:

<HOST NAME="myrouter" IP="192.168.0.254" REPORTED="1229761211" TN="28" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="0">
<METRIC NAME="uptime" VAL="3644" TYPE="uint32" UNITS="seconds" TN="28" TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/>
</HOST>

You can also specify the lifetime of the value, which will cause it to automatically be dropped from your gmond processes.

In addition to the gmetric command, there is are C and Python interfaces to gmetric. Additionally, if you want your own software to emit gmetric messages, there's an embedded gmetric library you can use.

My second question was, "What if a host is retired, is renamed, or is reconfigured?" If a host never again reports data about itself, the last state is still kept on every gmond and gmetad. To expire hosts that haven't spoken in a while, you should set the host_dmax value in gmond.conf to some value of seconds after which the host's state will be dropped. However, I am not sure the RRDs for this host will be cleaned up automatically. Host renaming probably requires that you rename the rrd directory holding the host's data if you wish to maintain historical data across the rename. If you reconfigure a host, deleting no-longer-used metric rrds is probably prudent. All of the above changes will likely require walking around in the ganglia rrd storage directory in your gmetad.

My last concern with ganglia was that the monitoring unit was a host. I think most often in services, not hosts. Thankfully, there are workarounds. Since you can spoof hostnames with gmetric, I decided to try something not-hostname-like to identify a service+host combo. I tried to spoof "service/hostname," but Ganglia uses this as the directory name and fails mkdir for not doing it recursively. Choosing another delimiter, comma, works fine:

Note: gmetric -S "foo/bar" will succeed, but gmetad will crash trying to write to that file path (mkdir not so smart). If you try this, you'll have to stop all gmond and gmetad instances then start them all again to clear the knowledge of the host named "foo/bar"

% gmetric -S "192.168.0.1:apache,aaa123" -n "apache errors" -v 5 -t uint32 -u "errors"
 spoofName: apache,aaa123    spoofIP: 192.168.0.1 

However, it appears that ganglia keys on IP as unique, so trying to add another entry of "192.168.0.1:mysql,aaa123" will appear in the web interface. Further testing revealed that the IP portion of the spoof is not validated. If we use the unique combination of "service,host" as both the IP and hostname, everything is peachy:

% gmetric -S "apache,aaa123:apache,aaa123" ...
% gmetric -S "mysql,aaa123:mysql,aaa123" ...

Pushing only apache-related data to the 'apache' host prefixes will help you organize your monitoring by service. This means you could easily view only apache data for a host, etc.

Ganglia makes me happy because I can worry only about giving it data. Adding a new data source only requires knowing how to get the data and feed it periodically to gmetric. After that, the data is automatically available in Ganglia's web interface. I fed (with gmetric) some fake data about mysql connections to see what happens, and this was the result.-

The wish-list of Ganglia features includes a talk of making it easy to provide custom graphs and views. Adding your own custom graphs requires, currently, hacking on the php code that presents the web interface. Glancing at the PHP powering the web interface doesn't make me cringe (easy to read!), so extending ganglia by adding your own views is probably reasonable.

Considering what I've found so far with Ganglia, the shortcomings mentioned here are easily worked around, and the potential benefits of less-painfully-configured data trending and monitoring seem quite good.

Oh, by the way, gmond works in Windows. The project has a cygwin binary that works by itself and can report data about a windows host. You can also use gmetric from windows hosts to report info about themselves.

Further reading:

• Ganglia home page
• Ganglia docs
• Ganglia Clusters and Grid Demo (web interface)
• Ganglia feature wish list

Day 8 - One-off graphs

You've got your nagios and cacti configurations all diligently tracking information for you. The data it monitors is available for viewing in graphs at your will, but what about the data it doesn't monitor?

You get a report that your apache servers are randomly serving errors and that the problem started last week. You don't have cacti watching this data, so you check the logs. The few-hundred-megs of logs are probably too much for you to eyeball, and seeing this data, now, in a graph, would help you out.

This report means two things: 1) Verify the report and fix the problem, and 2) add apache error servings to your monitoring system. What are your options for #1 and getting that data graphed now? Common sysadmin graph staples might include things like rrdtool, gnuplot, cacti, and others. Some of these tools are designed for recording and graphing data slowly over time and others require configuration changes or other complexities. It's possible you may be able to import historical data into your monitoring or trending system (cacti, etc), but if you don't know how, you have to graph it by yourself.

The path of least resistance is probably the best path when it comes to doing one-offs for visualizing or grabbing data. This means using the tool that requires the least amount of steps to go from data to graph with easy ability to iterate in case your graph output isn't helpful due to display decisions like scaling, etc.

Tools that help you do this include gnuplot, R, rrdtool, and Excel (or other spreadsheets that graph). These tools might help you manipulate the data before you graph it, but I'm going to assume that you've already got the data in some reasonable format (space, tab, comma delimited X and Y values).

We have apache access logs and want to see 500 error code trends. One approach might be to graph the ratio of 200s to 500s codes (200 is OK, 500 is internal error), or just graphing the 500s alone.

Making a useful graph depends much on how you aggregate your data. Do you aggregate on the hour, minute, 10 minute, second? You can go with your gut feeling, or you can take another approach. When gathering data, keep the data in the highest-precision format you have. In this case, we have data on the second precision.

# The '500' here matches the response code from apache logs
% egrep '" 500 [0-9]+ "' /b/access | sed -e 's/^.*\[//; s/\].*$//' | tee err500
01/Dec/2008:21:23:54 -0500
01/Dec/2008:21:24:08 -0500
01/Dec/2008:21:27:09 -0500
02/Dec/2008:05:23:34 -0500
08/Dec/2008:00:44:59 -0500
08/Dec/2008:00:45:55 -0500
< remainder of output cut >

# We count the instances per second by piping this output to 'uniq -c'
% uniq -c err500 > counts

If your graphing tool helps you make aggregation decisions such as "total 500s in an hour", then that's a help. Otherwise, you'll need to aggregate yourself before feeding your graph tool. RRDtool lets you do this by using the 'average' RRA and multiplying the value by the time interval. From what I can tell, gnuplot doesn't let you modify input data before graphing in a way that would let you aggregate values. R lets you do this easily as it's a statistics scripting language.

Data input for time series might require additional steps to convert the date into a value your graphing tool understands. Gnuplot accepts string time values and lets you specify the strptime(3) format string. RRDtool updates require times be specified in terms of unix epoch. R, from what I can tell, needs to be given numbers (like rrdtool). Excel hopefully has time parsing options, but I haven't tried.

Further, if your data doesn't have a point at every single unit of your graph, you will end up with odd-looking results when using lines to graph. This sways in favor of rrdtool since gnuplot and other tools that graph don't often accept this lack of data as OK. RRDtool has support for data points being 'unknown' and such and is much more drawn to time-series plotting.

Output is important too. Your graph is less helpful if the axes aren't readable; this means you need readable dates on your time axis. Both gnuplot and rrdtool allow you configure the X (time) axis labels and steps. It's difficult to do in R, from what I've tried and read.

For all the reasons above that help us see time-series data visualized most effortlessly, I would normally pick rrdtool. However, past experience has had me spend more time fighting rrdtool (read: pebcak) when I'm in a hurry, so I'll try gnuplot today. I fully confess in failing tonight trying to rush and re-learn rrdtool ;)

In gnuplot, you specify time as an input, from most any format, with:

set xdata time
set timefmt "%d/%b/%Y:%H:%M:%S"

If you want output to a file, use:

set terminal png size 580,300
set output "/tmp/apache.png"

The timefmt uses format strings specified by strptime(3). To graph the last year's worth of data in gnuplot (including the above xdata and timefmt lines):

set xtics rotate right
set xrange ["01/Jan/2008:00:00:00":"01/Dec/2008:00:00:00"]
set yrange [0:]
set format x "%Y/%b/%d"
plot "counts" using 2:1

Since 'uniq -c' (used above) outputs in the format 'count value' and our 'value' here is a timestamp for use with the x axis, we have to tell gnuplot to use the 2nd column for X and the count for Y.

This generates an ugly and not totally useful graph, because visualizing errors on that rate .

Changing from a seconds to another unit just requires some simple summation. Rounding the timestamp to whatever value (10 minutes, for example) and then doing another summation (uniq -c) on the output should be easy; any tool that supports strptime will help you, such as this small ruby script

If we sum errors by hour, the new graph gets a bit more useful, showing some days having very high error spikes compared to the average. As a note, since the output of strptime.rb is in unix epoch, I had to change the timefmt to "%s" and the xrange to '["1199145600":"1228089600"]'

Note: If you use gnuplot with it's default output device (don't run 'set terminal png') you get a useful GUI that you can zoom in and out of, which is pretty useful.

This is another case of having the right tools to do the job. I've used statistics tools like SAS before, and while writing this article today it feels like using such a tool to do simple, fast visualizations and analyses would be easier. It's possible R, Octave, or other math/stats tools provides this. On the other hand, I've never once heard of a sysadmin colleague using statistical tools, is this indicative of a problem?

Further reading:

Visualization periodic table

Neat periodic table showing lots of different visualization methods with examples

gnuplot

rrdtool

R's homepage