Day 20 - Github Gist

This article was written by Phil Hollenback (@philiph)

I assume everyone is familiar with the idea of a pastebin - a website for sharing text fragments with an emphasis on code fragments. Pastebins have been around since 2002, according to Wikipedia. They're an incredibly useful resource for sharing textual data and are something we, as sysadmins, need to do on an almost continual basis. However, there are several problems with some existing pastebin implementations:

• lack of command-line integration
• no version control
• no privacy settings

I recently came across a new (to me, anyway) alternative to the traditional pastebin: github gists. The following is a description of how gists work and how they differ from traditional pastebin clippings. I'll also describe some ways you can collaboratively edit gists with one or more people.

What's a Gist?

A gist is simply a text clipping with optional syntax highlighting, the same as you would find in any other pastebin. You can go look at some right now to get the idea.

So, why would you want to use this instead of a traditional pastebin? Pick a file (say, a perl script) and hold on to your socks:

$ gist test.pl
https://gist.github.com/737292

That's it! You just created a syntax-highlighted text clipping anyone on the internet can view.

Unfortunately, there is some up-front work to get this all set up. You can't just post anonymous gists to github.com like you can with some pastebins. I'll detail that setup info below. And, here's the really exciting part: there's an emacs script to automate all of this!

Initial Setup

As I mentioned, you have to have a github account to create gists (or to comment on existing gists). The good news is that's free and just takes a moment to set up. Once you have your account created, go to your account page and click on Account Admin. You will find your API token on this page. Take a moment to copy that down as you will need it to set up your command-line gist client.

You should also click on SSH Public Keys in the account settings page and upload your ssh public key. you're going to need this to edit gists shortly. Did I mention that gists are version controlled with git?

I'm assuming you have the git client installed for your linux or mac box already, if you don't have that go get it now as you will be using git a lot for all this. One thing that was not clear to me, initially, was how to set up your local git config for gist access. This is controlled by your ~/.gitconfig file, which will look something like this:

[user]
    name = <your name>
    email = <your email>
[github]
    user = <your github username>
    token = <your api token>

You can actually read and write from this file via git config on the command-line, like this:

git config --global github.user username
git config --global github.token blah

the gist command-line and emacs clients use this mechanism to read from your ~/.gitconfig.

Once you have this all configured, download and install the gist command-line client. I used the gem install gist install method which worked just fine. Verify your setup works by creating a gist, as above.

Now What?

At this point you've got a simple, command-line pastebin client which is a pretty useful thing. For example, suppose you want to demonstrate some code to someone on twitter. Instead of mucking with pasting your code into a regular pastebin website, feed your script directly to the commandcommand-st client. Right here you've got an url you can paste into your tweet. If the viewer of your gist goes through the small hoop of creating their own github account, they can leave comments about your gist too.

Don't worry, though - there's lots of other ways to use gists. For example, there's an emacs interface to gists!

Emacs Mode

The emacs interface for gists is gist.el. It supports mostly the same options as the regular command-line client with a few twists. For example, you can use gist-list to select from and open one of you public gists.

I've been using the emacs gist interface quite heavily to share gists with others. For example, if someone tells me 'check out my gist 741773', I can just hit <Meta>-x gist-fetch<RET>741773 to pop that gist right into an emacs buffer.

Unfortunately the emacs mode suffers from some glitches due to problems with ssl access in emacs. I had to hack on gist.el somewhat myself to get it working with Aquamacs on my mac. Thus while I'm pretty excited about gist.el, it's not really ready for primetime.

Markdown

In addition to plain text and programming language markup, gists also support Markdown. Actually they support Github Flavored Markdown, which includes a few small tweaks of the original Markdown language.

I assume most readers are familiar with Markdown, but if you aren't, it's a simple way to write structured ASCII text that can be easily turned into HTML or other document formats. The beauty of Markdown is it's completely readable as straight ASCII as well as HTML.

To force interpretation of your gists as markdown, use the .md file extension on the file you upload to create a gist. When you view your gist on github you will see it all dressed up with headers and bullets and everything.

Private Gists

By default, gists are public. This is the standard convention for pastebins - everyone can see what you post. This usually works just fine. However, if you want to protect your information, you can create a private gist. There are two differences between private and public gists:

1. public gists show up on the gist main page.
2. public gists use easily guessable sequence numbers, private ones use hash identifiers.

For #2, public gists have incremented IDs like 73962 while private gists use hashes like d17b2652f7896c795723. In practice, this makes it difficult to guess the ID (and URL) of a private gist. Note there is no real security here in the form of access controls - if someone obtains your private gist ID, they can access it. Thus, don't use private gists for passwords or other sensitive information.

However, private gists work just great for information you want to protect but isn't super critical. I would feel fine pasting config files as private gists, for example.

With the gist command-line client, use the -p switch to create a private gist, or use git-config to set your default gist posting mechanism to private. If you are going to use gist as a pastebin to share system information such as config files and scripts, you should probably use private gists by default. The emacs interface supports similar functionality.

Using git for Gists

As I mentioned earlier, gists are stored in a git repository on github. That means you can use them to collaborate on a documentation project. Here's the workflow:

1. Create a gist through web interface, cli, etc.
2. Give your friend Joe the url to that gist on github.
3. Joe visits that url and clicks 'fork' to get his own repository
4. Joe makes edits to his forked copy of your gist
5. Joe commits his changes to his repo, gives you his private clone url
6. cd into your local repository on your computer
7. Merge Joe's changes into yours with git pull <Joe's private clone url> master
8. commit your merged changes to your repo with git commit -a and git push

That's it! You're now collaborating with someone on a shared script, config file, markdown document, or whatever. Also, since this is a distributed version control service, your collaborator can always fork your gist and start modifying their own copy.

Remember that the gist web interface supports comments, so if you don't want to do a full collaboration with someone, they can always just leave gist comments instead (although commenters do need github accounts). Note that comments don't seem to be exposed in the git repository, unfortunately.

I've focused on single-file gists in this description, but note that gists can contain multiple files. You can add additional files via the web interface or by creating additional files in your local git repository. This is another important difference from traditional pastebins.

Why Should I Care About This?

As a sysadmin, I'm excited about this tool for a number of reasons. Mainly, I have a need to share scripts, config files, and the like with other sysadmins. Currently, that involves emails or traditional cut-n-paste pastebins. Neither of these solutions are very satisfactory.

What I want is a way to create public and private pastebins from the command-line and share those via a URL. I also want a way to mark up and collaborate on text files. Finally, it would be pretty handy if those files were automatically version-controlled and stored somewhere out on the internet for me.

Oh, also, that tool better not cost me anything, because I'm cheap and/or poor. Hey look, github gists support all those features! That's why I've started using gists instead of the old pastebins. The command-line and emacs integration are the real power of gists. Gists are a direct interface between your terminal and the cloud, all wrapped up in a sysadmin-friendly package.

Further Reading

• A good blog writeup about gists.
• The gist api.
• The gist homepage.

Day 17 - Monitoring with Xymon

This article was written by Kent Brodie.

When you're a sysadmin of several systems, you soon find yourself needing a centralized `dashboard' (ugh, I hate that term!) of sorts for your systems. That is, a central monitoring point where, with one or two screens, you can easily check on the overall health of ALL of your systems. And yeah, it helps when the same monitoring tool/suite notifies you 24x7 when something goes awry.

In this context, many product names are familiar: Nagios, Zenoss, Hyperic, OpenNMS, but one that I personally feel that gets overlooked too often is Xymon.

Anyhow, all of the above-mentioned products work well. Some are free, and some are mostly free unless you wish to purchase support or perhaps value add-ons. After trying most of them out, I settled on Xymon. Why? Because it's simple, lightweight, flexible, scalable, and yes, free. The others? They're all quite good, but I found them time-consuming to set up, and many of the screens display "too much" for my liking. There can indeed be "too much" of a good thing.

Historical note: Xymon is the current name of the project formerly known as Hobbit. Due to a nastygram the project lead got from the Tolkien estate some time ago, the project had to be renamed. You will notice however that portions of the original project name are still there (configuration file names, for example). So long as you know that Hobbit=Xymon, you'll be fine.

If you've never seen Xymon, you need to check out the online demo. I guarantee it will take you about 60 seconds (or less) to see how it works. Go to http://www.xymon.com. You'll see a simple screen with a few catagories and colored icons: green = good, yellow = warning, red = bad. The Xymon "home screen" is quite simple, and is one of the reasons I like this setup so much. Simplicity = elegance.

Navigating the display is simple. Start with the icon next to "systems." You'll see a few servers listed and lots of status icons. Click on ANY of the status icons for info. Click on a CPU icon. Once you've done that, scroll to the bottom. Check out the RRD historical graph at the bottom. And click on it. Whoa - lots of history and trend info. By the way, the online demo is actual live Xymon monitoring of a few servers belonging to the project's primary author.

After playing with the demo, you will either say "feh", or you'll say "whoa-- this is pretty powerful, yet - simple". For the latter folks: read on.

Setting up Xymon is simple. It's available as RPMs and source. I prefer source. Compiling it (follow the instructions) is straightforward. By the way, your distro may still use 'hobbit' as the package name. I'm not going to go into how to build and install Xymon, my goal is to describe a few of the key configuration files that make Xymon work. Note, as you read on, the simplicity of the setup. Xymon includes easy hooks into your local apache server.

The primary file that drives Xymon is bb-hosts (Why bb? Hobbit.. er.. Xymon is an active branch of the older, less capable "Big Brother" monitoring package). The bb-hosts contains the systems you want to monitor. The first entry is the Xymon host itself (and is required). The other two shown in the example below are two additional servers I want to monitor. (see bb-hosts(5)).

The basic format of this file is pretty straightforward: IP and host name. Other options are available as extra tests per host, such as checking web pages, verifying oracle is running, and so on. You can add the options later as you become familiar with how it all works.

# format is
# ip-address       hostname                # tag1 tag2 ...
192.168.1.10       xyhost                  # bbd apache=http://127.0.0.1/
192.168.1.1        system1
192.168.1.2        system2

Once you get rolling, you can eventually categorize and group the Xymon display page to your liking. For example, you can have all of your web servers in one sub-page and all database servers in another sub-page. You can mix and match, and hosts can occupy multiple sub-pages if you wish. It's all a simple matter of editing bb-hosts and following examples in there.

The next file that will become of great importance to you is hobbit-alerts.cfg. This file describes the notification actions to take when an event occurs.

SERVICE=conn
 MAIL my.email@my.domain RECOVERED FORMAT=TEXT

The entry above says, "Whenever ANY server is unreachable via a simple ping, send a detailed message to the email address listed". There's lots of flexibility here. You can have alerts sent when only certain servers fail, and ignore alerts for others. You can limit the times of day that alerts are actually sent, and so on. It's important to not have your Xymon environment "cry wolf" with too many false or chatty alerts, or everyone will start ignoring them. SMS texting is also available using out-of-band methods.

That's it! After setting up a hobbit server and configuring only a handful of simple text files, you now have basic monitoring. Without anything installed on clients, hobbit can remotely monitor things like uptime (ping test), http, ftp, ssh, smtp, and a few others. The power comes in when you install a little hobbit client on each server. Then, you'll have it all: CPU, memory, disk, processes (and more). You also have the ability to download/customize or even write your own tests. There's a ton of available modules you can add.

I highly recommend getting started slowly. After you have basic Xymon monitoring as outlined above, go ahead and build the client and install it on ONE of your additional servers above. Make sure to allow the xymon port (the default is tcp 1984) into the xymon server. When you check out the Xymon display, you will notice the server that has the client running will report a lot more information available for you to investigate.

When I set up a new server (we add stuff weekly), one of my configuration steps is adding it to the hobbit/xymon monitoring. On the Xymon server side, I add one text entry in the bb-hosts file. On the client side (the new server), I create a hobbit user account, grab the client tarball that I built earlier, and untar it in the hobbit user's home directory. With the final step of adding an init script, I'm done. Takes about a minute. The client is VERY lightweight, and gathers data using simple available tools like "top", "df", etc.

I like Xymon because it meets the "Gene Simmons" principle. It's simple, and it's effective. All of your sysadmin tools should be that way.

Kent can be reached at kbrodie at wi.rr.com if you have any questions.

Further reading:

• System Monitoring with Xymon on wikibooks

Day 18 - Logging Tools

Logging is good, right? There are lots of logging libraries for various languages to help you with logging events and behavior in your software.

You can log to files, log to email, log to the network, log from the network, and log to stdout, etc. As a software developer, you get to make a choice on where, how, and why you log. What if you're a systems administrator? What if the software developer chose to write everything to stdout without identifying information such as timestamps?

If the tools you are using all have configurable logging, you may not benefit from this article. If you have something without configurable logging, you're still in luck. There are a few tools out there that will help you turn output into good logging.

For this task, you have at least two options: multilog or logger.

logger

Logger is a helpful command-line interface to sending syslog messages. Since it uses syslog, you get all the beneficial configurability of syslogd.conf. The logger tool come standard on just about every unix-like OS (Linux, Solaris, FreeBSD, etc), and vary by feature. Here's an example of logger usage:

% logger "hello world"
% tail -1 /var/log/messages
Dec 17 22:20:13 snack jls: hello world

The default tag (-t flag) is $USER, or "jls" for me. Most programs use the tag for the program name and pid. If you run logger with no message, it reads messages from stdin:

% echo "hurray logging" | logger -t myprocess
% tail -1 /var/log/messages
Dec 17 22:21:58 snack myprocess: hurray logging

If you're on FreeBSD, logger can log to a remote host using logger -h <hostname>, meaning you can log to a remote host without modifying syslogd.conf, if you desire.

As a more useful example, here's how to make a cronjob log its output to syslog:

0 3 * * * rsync -av /data filer:/backups/data | logger -t "backup-data"

All of rsync's output will show up in syslog logs:

Dec 17 22:31:48 snack backup-data: sending incremental file list
Dec 17 22:31:48 snack backup-data: data/
< other output trimmed >

Configurability: You can choose where your log messages go (by file or host) with /etc/syslog.conf. Log rotation is done by using logrotate (linux) or newsyslog (freebsd), and most systems come with log rotation enabled by default.

multilog

The multilog tool is part of the daemontools suite of tools. Multilog's configuration is done completely on the command-line. With multilog, you can timestamp logs and even filter them to different logfiles. As a bonus, it supports logfile rotation.

Multilog logs to directories, not files. Each directory contains a few files: current, lock, and state. Other files that will show up here are rotated logfiles. A very simple example of multilog with timestamps follows:

% echo "Hello there" | multilog t ./log
% ls log
current  lock  state
% cat log/current
@400000004949f46513a31f5c Hello there

The wacky @4000....f5c is a timestamp in tai64n format. To convert tai64n to local, readable time, use the tai64nlocal command that comes with daemontools:

% tai64nlocal < log/current
2008-12-17 22:57:31.329457500 Hello there

The timestamps in the logs are very high precision, as you can see. And yes, tai64nlocal will convert to the correct timezone based on your environment (/etc/localtime or the TZ environment variable).

As a more complex example, we can have multilog log lines starting with 'ERROR' in them to another log directory in addition to the normal one. Here's how:

multilog t '-*' '+* ERROR*' ./errors '+*' ./log

This says to timestamp. By the way, 't' is required to be the first config option if you want timestamps. Then we use a few line selections: '-*' means deselect everything and '+* ERROR*' will select any line starting with error. This may not make sense, but the '* ' before 'ERROR' is because timestamps are added immediately, so the '*' matches the timestamp. Awkward, but it is what it is.

% alias mylog="multilog t '-*' '+* ERROR*' ./errors '+*' ./log"
% echo "Hello world" | mylog
% echo "ERROR something is wrong" | mylog
% cat errors/current
@400000004949f71c36d3a30c ERROR something is wrong
% cat log/current 
@400000004949f71c362af374 Hello world
@400000004949f71c36d3a30c ERROR something is wrong

Once you understand the configuration syntax for multilog, you can do some neat things with it as exampled above. Using multilog in the cronjob example above:

0 3 * * * rsync -av /data filer:/backups/data | multilog t /var/log/data-backups

And the output looks like this:

% head -2 /var/log/data-backups/current
@400000004949f9930274befc sending incremental file list
@400000004949f993038ece04 data/

# Let's make it readable:
% head -2 /var/log/data-backups/current | tai64nlocal
2008-12-17 23:19:37.041205500 sending incremental file list
2008-12-17 23:19:37.059690500 data/

One benefit to tai64n is that your log files are sorted simply by being prefixed with the timestamp (no sort(1) call necessary), so if you have gigs and gigs of logs, you could use binary search to very quickly find a small time range to investigate. Syslog's timestamp format does not sort properly, so you can't do this.

Logging is super useful. The tools described here will help you bring sane logging with timestamps and log rotation to any program that only outputs to stdout.

Further reading:

• multilog docs
• daemontools
• Syslog's network protocol is pretty simple

Day 12: Capistrano or Puppet?

I'm compelled to write about this subject today because of having received this question multiple times since sysadvent began.

Capistrano or Puppet? Both.

Puppet provides you with a way to specify a state your system should be in. Puppet's features will help you keep a machine in the same state. If someone hand-edits an apache config, you can have puppet automatically replace it with the correct one and reload apache, for example. Puppet runs on each of your servers.

Capistrano lets you describe what to do to a system or set of systems: Upload a file, run a program, restart a service, etc. Capistrano runs from your workstation and does work for you on remote systems.

So, why both?

Puppet needs a source of state information, and that source has to come from somewhere. If you always run on the bleeding edge of your configurations, you can feed the puppet master with your revision control system and use the state described in the head revision. Bleeding edges tend to be bloody for a reason. You could feed puppet with data from a branch of your revision control, too, and both not need capistrano for the feeding and not run on the head revision. You could deploy new state to puppet with Capistrano on a planned release schedule.

Puppet lets you specify that the 'httpd' package should be installed, and even what version. If you maintain your own package repository, you can control what version is installed (which you should). To upgrade the 'httpd' package, you could use Capistrano to upload new packages to your package repository and to deploy puppet manifests to keep 'httpd' automatically updated to whatever version you decide.

As an example, here's how the state management with puppet and capistrano might look for your apache configuration:

1. Modify httpd.conf in revision control, check it in.
2. Use capistrano to push the new state to your puppet masters.
3. Puppet will see the new state and apply necessary changes. [*]

[*] This will only occur automatically if you run puppet periodically (like through cron) rather than manually.

If the change was bad, you can revert the change in revision control and again use capistrano to push the new files.

You can use puppet and capistrano to do similar tasks, if you wish, but I find they are best suited to compliment each other. Let puppet focus on automated state maintenance and let capistrano help you do deployments of new packages and new configurations.

Further reading:

• Short comparsion of puppet and capistrano
• Puppet
• Capistrano

Day 8 - One-off graphs

You've got your nagios and cacti configurations all diligently tracking information for you. The data it monitors is available for viewing in graphs at your will, but what about the data it doesn't monitor?

You get a report that your apache servers are randomly serving errors and that the problem started last week. You don't have cacti watching this data, so you check the logs. The few-hundred-megs of logs are probably too much for you to eyeball, and seeing this data, now, in a graph, would help you out.

This report means two things: 1) Verify the report and fix the problem, and 2) add apache error servings to your monitoring system. What are your options for #1 and getting that data graphed now? Common sysadmin graph staples might include things like rrdtool, gnuplot, cacti, and others. Some of these tools are designed for recording and graphing data slowly over time and others require configuration changes or other complexities. It's possible you may be able to import historical data into your monitoring or trending system (cacti, etc), but if you don't know how, you have to graph it by yourself.

The path of least resistance is probably the best path when it comes to doing one-offs for visualizing or grabbing data. This means using the tool that requires the least amount of steps to go from data to graph with easy ability to iterate in case your graph output isn't helpful due to display decisions like scaling, etc.

Tools that help you do this include gnuplot, R, rrdtool, and Excel (or other spreadsheets that graph). These tools might help you manipulate the data before you graph it, but I'm going to assume that you've already got the data in some reasonable format (space, tab, comma delimited X and Y values).

We have apache access logs and want to see 500 error code trends. One approach might be to graph the ratio of 200s to 500s codes (200 is OK, 500 is internal error), or just graphing the 500s alone.

Making a useful graph depends much on how you aggregate your data. Do you aggregate on the hour, minute, 10 minute, second? You can go with your gut feeling, or you can take another approach. When gathering data, keep the data in the highest-precision format you have. In this case, we have data on the second precision.

# The '500' here matches the response code from apache logs
% egrep '" 500 [0-9]+ "' /b/access | sed -e 's/^.*\[//; s/\].*$//' | tee err500
01/Dec/2008:21:23:54 -0500
01/Dec/2008:21:24:08 -0500
01/Dec/2008:21:27:09 -0500
02/Dec/2008:05:23:34 -0500
08/Dec/2008:00:44:59 -0500
08/Dec/2008:00:45:55 -0500
< remainder of output cut >

# We count the instances per second by piping this output to 'uniq -c'
% uniq -c err500 > counts

If your graphing tool helps you make aggregation decisions such as "total 500s in an hour", then that's a help. Otherwise, you'll need to aggregate yourself before feeding your graph tool. RRDtool lets you do this by using the 'average' RRA and multiplying the value by the time interval. From what I can tell, gnuplot doesn't let you modify input data before graphing in a way that would let you aggregate values. R lets you do this easily as it's a statistics scripting language.

Data input for time series might require additional steps to convert the date into a value your graphing tool understands. Gnuplot accepts string time values and lets you specify the strptime(3) format string. RRDtool updates require times be specified in terms of unix epoch. R, from what I can tell, needs to be given numbers (like rrdtool). Excel hopefully has time parsing options, but I haven't tried.

Further, if your data doesn't have a point at every single unit of your graph, you will end up with odd-looking results when using lines to graph. This sways in favor of rrdtool since gnuplot and other tools that graph don't often accept this lack of data as OK. RRDtool has support for data points being 'unknown' and such and is much more drawn to time-series plotting.

Output is important too. Your graph is less helpful if the axes aren't readable; this means you need readable dates on your time axis. Both gnuplot and rrdtool allow you configure the X (time) axis labels and steps. It's difficult to do in R, from what I've tried and read.

For all the reasons above that help us see time-series data visualized most effortlessly, I would normally pick rrdtool. However, past experience has had me spend more time fighting rrdtool (read: pebcak) when I'm in a hurry, so I'll try gnuplot today. I fully confess in failing tonight trying to rush and re-learn rrdtool ;)

In gnuplot, you specify time as an input, from most any format, with:

set xdata time
set timefmt "%d/%b/%Y:%H:%M:%S"

If you want output to a file, use:

set terminal png size 580,300
set output "/tmp/apache.png"

The timefmt uses format strings specified by strptime(3). To graph the last year's worth of data in gnuplot (including the above xdata and timefmt lines):

set xtics rotate right
set xrange ["01/Jan/2008:00:00:00":"01/Dec/2008:00:00:00"]
set yrange [0:]
set format x "%Y/%b/%d"
plot "counts" using 2:1

Since 'uniq -c' (used above) outputs in the format 'count value' and our 'value' here is a timestamp for use with the x axis, we have to tell gnuplot to use the 2nd column for X and the count for Y.

This generates an ugly and not totally useful graph, because visualizing errors on that rate .

Changing from a seconds to another unit just requires some simple summation. Rounding the timestamp to whatever value (10 minutes, for example) and then doing another summation (uniq -c) on the output should be easy; any tool that supports strptime will help you, such as this small ruby script

If we sum errors by hour, the new graph gets a bit more useful, showing some days having very high error spikes compared to the average. As a note, since the output of strptime.rb is in unix epoch, I had to change the timefmt to "%s" and the xrange to '["1199145600":"1228089600"]'

Note: If you use gnuplot with it's default output device (don't run 'set terminal png') you get a useful GUI that you can zoom in and out of, which is pretty useful.

This is another case of having the right tools to do the job. I've used statistics tools like SAS before, and while writing this article today it feels like using such a tool to do simple, fast visualizations and analyses would be easier. It's possible R, Octave, or other math/stats tools provides this. On the other hand, I've never once heard of a sysadmin colleague using statistical tools, is this indicative of a problem?

Further reading:

Visualization periodic table

Neat periodic table showing lots of different visualization methods with examples

gnuplot

rrdtool

R's homepage