Day 12 - Scaling Operability with Truth

Written by Jordan Sissel (@jordansissel)

This article is about separating your configuration model from your configuration inputs. You might have the same service (like frontend, loadbalancer, etc) in multiple environments, but versions, accounts, members, etc, may change between them. The 'scale' part, here, is that having inputs separate from models helps to scale operability and sanity. I use Puppet features to show how this is done, but this idea is applicable to any config management tool.

Puppet and other configuration tools help you describe your infrastructure. Puppet's model is resources: files, users, packages, etc. Each resource has attributes like package version, file path, file contents, service restart command, package provider (gem, apt, yum), user uid, etc.

Many of my puppet manifests used to look a bit like this:

class nginx::server {
  package {
    "nginx": ensure => "0.8.52";
  }
  ...
}

To upgrade versions, I would just edit the manifest and change the version, then push out the change.

Static manifests don't scale very well. Puppet has support for environments so you can use different manifests (say, a different nginx::server class) in different situations. You could also do it all in one like this:

class nginx::server {
  package {
    "nginx":
      ensure => $environment {
        "production" => "0.8.52",
        "testing" => "0.8.53",
        "dev" => "latest",
      };
  }
}

The variable, $environment, comes from what puppet calls a 'fact.' Facts such as cpu speed, ip addresses, chassis make/model, virtualization source, etc, are provided to help you conditionally tune your infrastructure. Facts come from a puppet tool called Facter, which means facts are essentially an external input for puppet.

This model of external inputs is a good one. Using facts, you could conditionally install things like Dell OpenManage, or template service configs based on ram or cpu counts. Combining the inputs (facts) with your model (puppet resources) helps you steer puppet into doing the correct thing for each machine it is run on. Very useful!

Back to the nginx manifest above. The problem with the above puppet example is that it doesn't scale very well. Each time you add a new environment (a new testing environment, some one-off dev cluster, etc), you have to edit the manifest (your infrastructure model). For each 'environment' condition, you have to update it with the new environment-specific value, which could mean editing just about every file in your puppet manifests - and that's sucky and error-prone work.

The solution here is to follow the facts example and allow yourself a way to specify your own inputs - what I would call, "truth," Truth is very similar to facts. It is just more human-driven, and I only use a different, but similar, term for the sake of identifying source. I use "truth" to refer to human-originated inputs (such as "what package version to install"), and I use "facts" to refer to machine-originated information (such as "number of cpus on this system").

For the above puppet manifest, you could write truth in separate manifest that defines $nginx_version based on $environment:

# truth.pp
case $environment {
  "production": { $nginx_version = "0.8.52" }
  "testing": { $nginx_version = "0.8.53" }
  "dev": { $nginx_version =  "latest" }
}

# modules/nginx/manifests/server.pp
class nginx::server {
  package {
    "nginx": ensure => $nginx_version;
  }
}

This is better, but ultimately, setting variables with conditionals (case statements, etc) is more code than you need - in Puppet 2.6.1, a solution to this problem was included: extlookup(). (It existed prior, but it now officially ships with puppet)

This new function lets you separate your configuration inputs from your infrastructure model. An example of the above manifest using extlookup:

class nginx::server {
  package {
    "nginx": ensure => extlookup("package/nginx");
  }
}

You configure extlookup in puppet by setting some variables, here's an example:

$extlookup_datadir = "/etc/puppet/manifests/extdata"
$extlookup_precedence = ["nodes/%{fqdn}", "environments/%{environment}", "common"]

The values in %{ ... } are evaluated with each node's facts. This lets you put all production environment data in /etc/puppet/manifests/extdata/environments/production.csv.

With the above configuration, I can easily add per-node and per-environment package versions for nginx. If all else fails, I can provide a default package version in the 'common' setting above. You can also easily extend the configuration to include more sources without modifying your infrastructure model.

The data format extlookup uses currently is csv files. It's basically a bunch of key-value pairs you can query with a precedence order. For example, the above config would look in nodes/somehostname.csv first, which would allow you to override environment-specific configuration values per host. If nodes/somehostname.csv file does not exist or if the requested value is not present in that file, extlookup will fall to the next file in the list. You can also specify default values in extlookup per call, but that is outside the scope of this article.

Building on the examples above, here is what your extlookup files would look like:

# /etc/puppet/manifests/extdata/environments/production.csv
package/nginx,0.8.52

# /etc/puppet/manifests/extdata/environments/testing.csv
package/nginx,0.8.53

# /etc/puppet/manifests/extdata/environments/dev.csv
package/nginx,latest

This scales in meaningful ways. Common cases:

• You want to clone production:

  % cp production.csv productionclone.csv
  # Or, accept 'testing' as good, and ship it to production:
  % cp testing.csv production.csv
  

• A security vulnerability makes you want to know what nginx versions you are installing:

  % grep 'package/nginx' */*.csv
  

• Tune settings per machine, per environment, etc.

• All of your truth inputs are concentrated in one place. This has an additional benefit of allowing non-puppet users (in this example) to upgrade or tune values without needing puppet knowledge.
• If you author puppet modules, you can release modules with extlookup support and document the values used. This allows users of your module to be happily ignorant of the internals of your module.

Point is, once you separate your model from your inputs (facts, truth, etc), you become more flexible and agile. I use extlookup for:

• package versions. Of note, for developer systems, I set some package values to "absent" so developer activity won't be overwritten by puppet trying to deploy our internal apps.
• application flags and tuning. For production, we want to do backups to Amazon S3. For random development systems, we don't. In general, I use this for helping generate production application configs.
• syslog configuration (what remote syslog host to ship logs to, per deployment)
• database information (database name, credentials, etc)
• some dns automation (what certain internal CNAME records point to)
• nagios contact configuration (example: notify pager in prod 24/7, only during daytime for staging, but never for development)

In closing, separating your model from your inputs increases the testability and maintainability of your infrastructure automation. Further, combining human input source (truth) with machine inputs (facts) can help you tune your infrastructure for any situation. Having a separate truth input allows you to abstract tunables in your infrastructure and opens up maintenance possibilities to non-experts (like allowing developers or QA folks managing their own clusters), which can help improve non-sysadmin throughput by allowing others to manage tunables you define. That helps you stay sane and uninterrupted, and it helps the business.

Further reading:

• Puppet facts with Facter
• Chef facts with Ohai
• Complex data and Puppet

Day 6 - Aggregating Monitoring Checks for Better Alerts

This article was written by Jordan Sissel (@jordansissel)

Chances are that your infrastructure has more than one machine performing the same role - frontend (webapps, etc), backend (databases, log collectors, etc), middleware (message brokers, proxies, etc), support services (config management, etc). Horizontal scaling means that you scale by adding more server resources, right?

Monitoring a horizontally scaled system can often lead to unintended noise. You might have 10 web servers, but 3 are broken, so you get 3 alerts. Maybe you monitor ssh health on all 50 of your servers, and someone pushed a bad config or there is a bad network connection, and now all 50 servers are alerting you about failures. Further, I believe that monitoring alerts (things that page you) should generally be business-oriented, not single-metric-and-single-server-oriented.

Even if you aren't running systems with duplicate roles, it often makes sense to collect multiple individual health checks into a single alert.

This calls for aggregating your data. Benefits of aggregation are increased alert value while decreasing alert noise. Noisy monitoring systems often result in yourself (and others) unconsciously training to ignore that system - don't let your monitoring servers cry, "Wolf!". Further, alerting on aggregates means you can do cool things like only alerting if, for example, more than 5% of things are failing - assuming that's safe for your business.

If you could keep the existing checks you have, not alert on them, but add aggregate checks that cause alerting, you could maintain your current fine-grain monitoring while improving alerting in both noise level and business relevance! Sounds like win-win.

There are some solutions for this already, like check_multi. but I didn't want to run additional or duplicate checks. Why? Some checks may take a long time (like a complex Selenium test), or may be resource-intensive to run (like verifying a large backup), or may not scale if I have to run 200 of them to answer one aggregated health check.

So, I took a different approach. Nagios already records state in a file on disk, so we can parse that file and produce an aggregate check. Lucky for me, there are already tools that parse this format, so most of the work is done already (nagiosity, checkmk_livestatus, and ruby-nagios). I picked ruby-nagios, for no particular reason. It didn't require much hacking to add what I needed. The result is nagios-manage and a tool: check_check.rb (comes with the nagios-manage rubygem).

Notes: Currently, this tool assumes a few things about what I want:

• Skips checks that have notifications disabled
• Skips checks that are marked in downtime
• Only uses the hard state. If it is in soft critical, for example, it will assume the previous hard state.

The above constraints let me aggregate other nagios-state data into health that is deemed important - if one server is down for maintenance or has a known failure, I don't want to alert about it.

Examples:

# Summarize all checks for all hosts:
% check_check.rb
OK=110 WARNING=0 CRITICAL=0 UNKNOWN=0 services=// hosts=// 

# Summarize solr checks, when one check is failing:
% check_check.rb -s solrserver
OK=27 WARNING=0 CRITICAL=1 UNKNOWN=0 services=/solrserver/ hosts=// 
Services in CRITICAL:
  frontend1.example.com => solrserver client tests

The result in nagios looks something like this:

By the way, the no-argument invocation is super useful for, from the command line, checking the overall health of everything nagios monitors.

For the solrserver check above, I would create one single check that would run the above check_check.rb command and alert appropriately. This saves me from having 28 potentially-failing-and-alerting checks. Further, if I get an alert about 1 critical check, I still have the further fine-grained alerting per server, so I can easily drill into what is failing.

Summarizing: I still have monitoring doing fine-grained, individual checks, like cpu usage, network activity, server behaviors, etc, but individual checks do not cause alerts themselves. This means setting the contacts for your individual checks to something that won't alert you. Instead, I use check_check.rb to aggregate results to reduce noise and to collect relevant data - alerting only on important failures in a way that doesn't flood the pager and helps me better indicate business-service problems to those on call.