December 7, 2014

Day 7 - What System Administrators Can Learn From Community Management

Written by: Daniel “phrawzty” Maher (@phrawzty) & Jonathan Clarke (@jooooooon42)
Edited by: Yakira Dixon (@yakiradixon)

Introduction

At first blush it might not seem like there are any parallels between system administration and community management, but the two fields have much more overlap than you might think! Before we begin, let’s define community management briefly, and to do so, I’m going to lean on this this excellent post by Deb Ng. It’s not marketing, nor social media, nor tech support, though these things can be involved. A community manager is a strategist, a content creator, and above all, a communicator. In this article, we will explore a handful of key principles from the community management world and see how they apply to system administration.

It’s important to realise that both fields are concerned with systems, which is to say, “an assemblage or combination of things or parts forming a complex or unitary whole” (source). In the case of sysadmins, one normally thinks of the relationship between nodes in a cluster, or clusters in a farm, or farms in a data centre, or - well, you get the idea. For community managers, one normally thinks of the relationship between members of a forum, or forums in a site, or sites in a community, or - well, here again, you get the idea. In both cases, the relationships between the individual elements of the whole can be complex, and often require specialised tools and years of experience to get a handle on.

Managing groups at scale

Scale is an important concept in community management. What works for managing a group of two people might not work for a group of 20 or 200 people. It’s generally easier to discuss a topic in a small group and achieve consensus when there are mutually-shared interests. Small groups can function directly together since the number of individual voices is low and the space needed for them to work is commensurate with their size. Logistically speaking, a small group can make use of pretty much any communication platform that happens to be convenient - a common list of people to CC when emailing and a Google document might well be enough.

Contrast that with managing a community of, say, 50 people - that’s a big list of people to keep tabs on every time you need to write an email (not to mention the amount of noise that might get generated with an errant reply- all), and as anybody who’s ever tried to play a fast-paced game of multiplayer text editing knows, that’s a lot of cooks in the kitchen when it comes to a shared document. Even at this modest scale, some sort of more purpose-built tooling is required, such as a proper mailing list or forum site. This necessity is borne from two important effects of larger groups which, for our purposes, we can refer to as quantity and quality.

Quantity is straightforward: more people means more conversations! A handful of emails per week is trivial for any one person to deal with, but multiply that by a factor of ten or 100, and the trickle becomes a raging torrent. The more information that’s being generated, the more difficult it becomes for any one member of the group to process it effectively.

Quality refers to the nature of those conversations, and in particular, their level of diversity. Each person in the community will bring their own particular histories, viewpoints, cultural perspectives, and so forth to the group. Good tooling can make all the difference when trying to manage such a heterogenous system. A straightforward example of this is mature forum software that has conversational threads, thematic areas, and a robust back- end for gathering statistics and metrics about usage.

From here it shouldn’t be difficult to make the mental leap to system administration. Two servers sitting on your desk are pretty straightforward to deal with - even if they’re totally different, there’s only two of them, so you probably don’t need any special infrastructure to manage them. Scale that up to a dozen servers, however, and the situation becomes much more complex. The dual effects of quantity and quality are instantly applicable. At even relatively small scales the importance of a coherent configuration management tool becomes evident: something that allows the different nodes in the system to be organised as operational or thematic groups, and with a robust back-end for gathering statistics and metrics about usage. Sound familiar?

We’re going to come back to scale later, but for now, let’s move on to something a little bit more esoteric.

Writing a mission statement

I know what you’re thinking: “mission statement? That’s just corporate-speak nonsense!” And yes, that can certainly be the case, but it doesn’t have to be. In fact, a good mission statement is essential for any community, and explicitly stating it can help to crystallise the community in powerful ways.

Most organic communities assemble around an initial idea or principle. While this shared element is enough to form the community, it’s rarely enough to maintain it, and almost certainly insufficient for real growth - the secret sustainability sauce is made up of the values that original germ represents. In other words, people come for the brilliant spark, but they stay for the ideas generated from the spark. A mission statement is a way to codify and present both.

Sound a little high level? Consider that at some point in your life you’ve probably come across an open source project that was little more than a few hundred lines of code and a README. But something about this project made it different from the others. The documentation, ersatz though it might have been, clearly elucidated the problem and how this project acted as a (potential) solution. This simple addition to the repository made all the difference in the world because it helped you to understand how the project could fit your particular situation. Even without a deeper analysis of the code in question, your interest was piqued and your desire to learn more was set into motion. What happened there? Believe it or not, what captured your attention was, fundamentally, a mission statement.

Most established companies have mission statements. These are organisation- wide screeds that may or may not have any direct relationship with the work that any one contributor effects within that organisation. Is it possible to bring the granularity level down a notch or two, and if so, what might the benefit be? Could it be useful to a system administration team?

Imagine you’re just starting on a new project: a simple but powerful collection of scripts to manage virtual machines that’s going to make everyone’s day-to-day work much easier. Right now, even before you’ve begun writing any code, is the perfect time to write a mission statement. Start with something simple like “I’m fed up of typing in a dozen long and complicated commands every time I need to migrate a VM in our infrastructure, especially in the middle of the night, when on call. I intend this to be a small wrapper tool that can do normal migrations for us by typing in just one command. This should reduce the amount of time we waste on manual migrations and help to eliminate typing errors”. This brief, simple statement encapsulates both the problem and the solution, clearly defines the scope of the project, and describes both the initial idea germ and - critically - the parameters for success.

There are numerous benefits to drafting your mission statement at the earliest possible stage. Crucially, by acting as a sort of combination compass and mold, it acts as a guide to keep the contributors focused (even if you’re the only one), which helps to nip things like feature creep in the bud straight away. Keep it short and don’t overthink it. Much like setting up some new test infrastructure, or working on a new program, the best thing is to start small and iterate over time as necessary. Finally, writing a mission statement should be easy; if you’re embarking on a project, then you know best what problem you’re trying to solve, and at those early stages is when you’ll have the most clarity about it. If for no one else, write the mission statement for your future self - a few weeks down the line you may have gone so deep you completely forgot the original problem you were trying to solve!

A good mission statement can also be used to generate excitement which, as we’ll see below, can be a powerful tool for getting things done.

Encouraging involvement

A key principle of community management is actively encouraging contribution and involvement from the community. A good community manager will generate excitement and foster enthusiasm. Cultivating a positive emotional connection to the community will make members more loyal and encourage them to participate even more - a virtuous cycle that in turn makes the community even stronger. That said, while a general feeling of excitement is, like that initial idea germ, enough to create a spark, it’s insufficient to maintain the community. The excitement needs to be backed up by concrete measures to encourage active involvement.

There is a fairly well-known philosophy in the open source world called “Release early, release often” that was popularised by Eric S. Raymond in the late 90’s. The essence of the philosophy is that sharing versions of a software project early and often will ultimately improve the overall quality of the software. This happens because beta-testers and early adopters can have fun playing with new features while contributors can learn the codebase by fixing obvious bugs - that early feedback will help influence development in useful ways. In fact, as long as they’re not fatal, it can be helpful to leave some bugs in there deliberately! Providing low hanging fruit can enable an enthusiastic contributor to jump into the project by giving them the chance to fix that bug themselves.

Looking at this from a systems perspective, this is basically a process with a feedback loop, and by releasing often the loop is accelerated. In other words, excitement is generated by rapid development, and the rapid development is encouraged by excitement - it’s almost too easy!

As a system administrator, while you may certainly be leading up a project for which the rapid release model is applicable, that won’t always be the case. That’s no reason not to adopt this principle, however; by actively encouraging your co-workers to get involved, you help to build a trust relationship that will pay dividends down the line. Consider the development vs. operations silo for example - how nice would it be if the two groups actually worked together on projects right from the start?

So how do you generate excitement? The easiest thing is to talk to your co- workers. Write a mission statement - or, better yet, draft one in collaboration with some potentially interested parties. Ask questions, discuss ideas, and continuously act on that feedback. Share design documents, proofs of concept, and initial implementations. Don’t be shy about unfinished releases or weird bugs, and don’t let perfection be the enemy of good - the important thing here is that both the features and the bugs are clearly explained.

Whatever the scope of your next project, try sharing the idea with your co- workers (or whomever) before you get started and see how they react. Chances are, you’ll get their support a lot more easily that by showing them hundreds of lines of script that don’t yet solve the whole problem - and they probably won’t nitpick about your chosen scripting language or variable naming while reading the mission statement either.

Of course for others to be able to fix bugs in your code, they need to know how to get that bug fix back into the code. Which brings us to our next topic"¦

Establishing community guidelines

A simple search with the term “community guidelines” will reveal hundreds of thousands of examples from across the web. They encompass, either implicitly or explicitly, communities of just about every conceivable size on more or less every topic there is. It’s difficult to find a legitimate site that doesn’t have one - but, what are they? Simply stated, they are the basic tenets that frame the social contract between all of the members of the community, and they serve as ground rules to help that community to interact well. For example, an increasingly popular type of guideline is the conference code-of-conduct, which most conferences have now to help ensure that respectful and inclusive behaviour prevails.

Responsible community management is, in many respects, predicated on the existence of community guidelines. It’s one of the most important tools that leaders can use to act decisively and consistently.

When technical folk collaborate, be it in writing code or in administrating a server, we all tend to do things slightly differently from one another. How so? A simple question: where do you put custom binaries on a server?

Right.

Now, chances are you’re thinking “I don’t need to write out guidelines - we’re a tiny group”. You may well be right - your group probably can function just fine, for now; however, consider when a new person is joining your team. They will need to understand “how to do stuff around here”. Thinking further, the future could include a larger expansion, perhaps open sourcing a tool you wrote, or your company being acquired or merging with a larger one. In these cases, having planned ahead and written down the key points on how to work together could be a life saver - for both yourself and the new arrivals.

As with everything, the key is to start simply: put a minimum viable set of guidelines together and then iterate as necessary. Some good starting points might include things like “all code must live in a given version control system.” That’s a simple, effective guideline that is both easy to follow and hard to misinterpret. Stipulate the conventions up front: paths on file systems, tabs vs. spaces for indentation, and so forth. That’s a great start.

Next, explain how to contribute; this could be by sending a pull request on GitHub, a patch by email, or whatever. The important thing is to set up a consistent framework. Elucidate expectations for responses to contributions, too - it’s better to say something like “I only review patches on Mondays, so please be patient” than to leave an enthusiastic first time contributor hanging for six days. Examples of such standards documents (that have open source licenses) include Google’s style guide, or for a more simple example, Rudder’s technical guidelines.

These contribution guidelines obviously tie into lowering the barrier to entry to your project too (see previous section). The underlying goal is to facilitate collaboration by avoiding pointless debates about which indentation style to use, when the project could be accepting contributions and moving forwards. There are no hard and fast rules governing the content or structure here, but as a general rule the idea is to be concise - use clear, brief language that expresses the core principles accurately.

Similar guidelines can, and should, be drawn out for social interaction. Don’t fall into the trap of defining complex rules that are hard to follow and, most importantly, assume good intentions by default. Great relationships are built on trust, and the best way to get trust is to give it in the first place. There is really no one-size-fits-all set of rules and each community will grow around the shared values that have particular meaning to them; however, it is very important to point out what is not OK. Again, exact details here depend on who you are as a community but make sure that as a bare minimum you explicitly forbid any kind of discriminatory behaviour or harassment.

Last but not least, remember that as one of your community’s leaders - however big or small that community is - you should be setting the example through action. Respect your own conventions. Welcome newcomers. Communicate clearly, consistently, and often. Be mindful that your words carry weight, and choose them deliberately and carefully when you write every email, blog post, or README.

Resources

The web is full of resources about community management - if you’re curious to learn more, some good starting points (for system administrators) include the Producing Open Source Software book, Eric S. Raymond’s The Cathedral and The Bazaar, and The Mozilla Manifesto. Dawn Foster’s blog also has some great insight on managing a community from a sysadmin perspective.

December 6, 2014

Day 6 - Debugging for Systems Engineers

Written by: Pris Nasrat (@nasrat)
Edited by: Ben Cotton (@funnelfiasco)

Overview

I’ve mentioned before that my favourite prior sysadvent articles is Down The Rabbit Hole.

This year amongst some of the reading I’ve done, I really enjoyed the articles from Julia Evans on strace. I wanted to write something for sysadvent that would be interesting but focussed on debuggers. There isn’t enough space here to give a full debugger tutorial but instead wanted to give some cases when I reach for a debugger and what for, with some specific tips and examples thrown in. If you’re already an expert at using debuggers you can probably stop reading now.

Many of us are familiar with some of variety this quote:

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it? - Brian Kerninghan

learning debugging techniques for programs you haven’t written is a skill I’d recommend systems engineers/administrators practise.

Forms of debugging

Print statement debugging

The most effective debugging tool is still careful thought, coupled with judiciously placed print statements - Brian Kernighan

If you spend any time on the golang-nuts list you’ll see that using gdb with highly concurrent runtime such as go is problematic, and certainly the latter part of the above quote is often recommended. If you have the luxury of being able to rebuild and rerun your application or service - such as when developing it then print debugging is a valuable part of your too chain. If you’ve written a helloworld program in a language you know at least one way of getting your program to print information out to you.

Program tracing

Julia has a fantastic blog post on debugging your programs like they’re closed source covers using strace and friends to do black box analysis using operating systems tools.

Using a debugger

This is where I want to share some of my knowledge. …

Setup

Lets build a container for our debugging experiments and launch a shell in it. I’m using Docker to make it easy for you to follow along with the same environment, lets create a directory for this with the following Dockerfile

FROM fedora:20

RUN yum -y update && yum clean all
RUN yum -y install gdb yum-utils && yum clean all
RUN debuginfo-install -y coreutils && yum clean all
RUN mkdir -p /tmp/a && touch /tmp/a/foo

Most distributions ship with stripped and optimized binaries, so to get some more meaningful traces we need to install some additional packages for debug symbols. Debian and Fedora derived distros (and probably others) include ways to get these extra packages:

$ docker build -t gdb .
$ docker run -ti gdb /bin/bash

Running a program through the debugger

Now off lets setup to launch a program with some args in the debugger:

gdb  --args /bin/ls /tmp/a
GNU gdb (GDB) Fedora 7.7.1-21.fc20
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bin/ls...Reading symbols from /usr/lib/debug/usr/bin/ls.debug...done.
done.
(gdb)

As you can see gdb is quite verbose by default so following invocations of gdb will use the quiet flag -q to suppress the banner and license information.

All that has happened is so far is that gdb has loaded the symbols from the file and the additional debuginfo, we’ve not run the program yet. Lets just do that now by typing run - note that (gdb) is the prompt output from gdb.

(gdb) run
Starting program: /usr/bin/ls /tmp/a
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
foo
[Inferior 1 (process 28) exited normally]
Missing separate debuginfos, use: debuginfo-install pcre-8.33-7.fc20.x86_64 xz-libs-5.1.2-12alpha.fc20.x86_64

So far so good - we see it starts the program, prints some info on how it is running and then the foo is the output of our ls command.

To make it clearer maybe we should have chosen ls -l lets change the state of the debugger to update the arguments

(gdb) show args
Argument list to give program being debugged when it is started is "/tmp/a".
(gdb) set args -l /tmp/a
(gdb) show args
Argument list to give program being debugged when it is started is "-l /tmp/a".
(gdb) run
Starting program: /usr/bin/ls -l /tmp/a
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
total 0
-rw-r--r-- 1 root root 0 Dec  3 15:44 foo
[Inferior 1 (process 32) exited normally]

Exit gdb using quit or Ctrl-D.

Stop the world

Still in our container lets start gdb again this time explicitly using ls -l:

gdb -q --args /bin/ls -l /tmp/a Reading symbols from /bin/ls...Reading symbols from /usr/lib/debug/usr/bin/ls.debug...done. done. (gdb)

Running a program straight through isn’t that interesting, if you’ve done basic C you’ll know that the main method is the entry point to our program. We’re going to tell gdb to create a breakpoint to pause the program when it hits the main method.

(gdb) break main
Breakpoint 1 at 0x402c60: file src/ls.c, line 1242.

Using gdb you can abbreviate commands until they become ambiguous. We could have used b main. With the breakpoint in place lets run again:

Starting program: /usr/bin/ls -l /tmp/a
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, main (argc=3, argv=0x7fffffffe798) at src/ls.c:1242
1242    {
Missing separate debuginfos, use: debuginfo-install pcre-8.33-7.fc20.x86_64 xz-libs-5.1.2-12alpha.fc20.x86_64
(gdb) 

The above tells us we’ve hit the breakpoint at line 1242 which has a single { , but not much else. Lets see some more context by using the list command:

(gdb) list
1237        }
1238    }
1239
1240    int
1241    main (int argc, char **argv)
1242    {
1243      int i;
1244      struct pending *thispend;
1245      int n_files;
1246
(gdb)

Repeated use of list will show us the next few lines, with gdb just hitting return will repeat the previous command - eg if you hit return after the above listing we see the next few lines.

(gdb)
1247      /* The signals that are trapped, and the number of such signals.  */
1248      static int const sig[] =
1249        {
1250          /* This one is handled specially.  */
1251          SIGTSTP,
1252
1253          /* The usual suspects.  */
1254          SIGALRM, SIGHUP, SIGINT, SIGPIPE, SIGQUIT, SIGTERM,
1255    #ifdef SIGPOLL
1256          SIGPOLL,

Even though we’re paging through the source code we’ve still not executed it - if you get confused you can always ask gdb where you are:

(gdb) where
#0  main (argc=3, argv=0x7fffffffe798) at src/ls.c:1242

If you’re not familiar with C the argc is the count of arguments and argv is an array of strings. We can use gdb’s print command - which abbreviates to p to inspect these.

(gdb) print argc
$2 = 3
(gdb) p argv[0]
$3 = 0x7fffffffe97f "/usr/bin/ls"
(gdb) p argv[1]
$4 = 0x7fffffffe98b "-l"
(gdb) p argv[2]
$5 = 0x7fffffffe98e "/tmp/a"

For now ignore the $2 = part and just focus on the values. For argc which is an int we just get the numeric value. Using the size we can look at the three strings forming argv using array access.

ls is quite a large program - for now lets tell it to continue executing:

(gdb) cont
Continuing.
total 0
-rw-r--r-- 1 root root 0 Dec  3 15:44 foo
[Inferior 1 (process 50) exited normally]

Now we know a little on how to set a breakpoint and get some basic information on variables in the program. Lets try another program - setting a breakpoint in another method. For now I’m choosing to use cat.

[root@7dffa749b76c /]# gdb -q --args /bin/cat /etc/passwd
Reading symbols from /bin/cat...Reading symbols from /usr/lib/debug/usr/bin/cat.debug...done.
done.
(gdb) break simple_cat
Breakpoint 1 at 0x4026b8: file src/cat.c, line 177.
(gdb) run
Starting program: /usr/bin/cat /etc/passwd

Breakpoint 1, main (argc=2, argv=<optimized out>) at src/cat.c:730
730           ok &= simple_cat (ptr_align (inbuf, page_size), insize);

So we stop before we execute the simple_cat method

(gdb) list
725              || show_tabs || squeeze_blank))
726         {
727           insize = MAX (insize, outsize);
728           inbuf = xmalloc (insize + page_size - 1);
729
730           ok &= simple_cat (ptr_align (inbuf, page_size), insize);
731         }
732       else
733         {
734           inbuf = xmalloc (insize + 1 + page_size - 1);

Rather than continuing the exection we are going to tell the debugger to execute the simple_cat method then stop using the next command. Some debuggers call this “stepping over”.

(gdb) next
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
Breakpoint 1, main (argc=2, argv=<optimized out>) at src/cat.c:730
730           ok &= simple_cat (ptr_align (inbuf, page_size), insize);
(gdb) p ok
$1 = true

As you can see we’ve not moved lines but have executed the subroutine - which output the file and returned. We can see the value of ok set. Stepping over is useful when you want to follow the flow of something but don’t care about what it is calling.

Type cont to continue execution.

Going deeper

In addition to stepping over, most debuggers allow you to follow the code into the method by stepping into. In gdb the command is step

gdb -q --args /bin/cat /etc/passwd
Reading symbols from /bin/cat...Reading symbols from /usr/lib/debug/usr/bin/cat.debug...done.
done.
(gdb) break simple_cat
Breakpoint 1 at 0x4026b8: file src/cat.c, line 177.
(gdb) run
Starting program: /usr/bin/cat /etc/passwd

Breakpoint 1, main (argc=2, argv=<optimized out>) at src/cat.c:730
730           ok &= simple_cat (ptr_align (inbuf, page_size), insize);
(gdb) step
simple_cat (bufsize=65536,
    buf=0x60f000 "root:x:0:0:root:/root:/bin/bash\nbin:x:1:1:bin:/bin:/sbin/nologin\ndaemon:x:2:2:daemon:/sbin:/sbin/nologin\nadm:x:3:4:adm:/var/adm:/sbin/nologin\nlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin\nsync:x:5:0:sync:/"...) at src/cat.c:177
177       if (n_read == 0)

When you’ve entered a function cont acts as a sort of step return where you run all the code in the method and stop at the return point.

(gdb) cont
Continuing.
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin

Breakpoint 1, main (argc=2, argv=<optimized out>) at src/cat.c:730
730           ok &= simple_cat (ptr_align (inbuf, page_size), insize);

Getting to know your codebases

I learned this technique from a friend as an early thing I do when learning a new service or application I’m going to work on or support.

Requirements:

  1. I make sure have a US Letter/A4 pad which I usually make sure is landscape, some pencils/pens next to me.
  2. Next step is to learn how to checkout, build and run the application locally.
  3. Choose a point to dive in for - eg for a command line app this might be main()

Then I use step-debugging to follow through and sketch out the main conceptually flow of the program, and dive in to parts I’m interested in. This works particularly well within IDE debuggers with the source code such as Eclipse or Intellij (or gud mode of emacs). Where you can set a break from the sourcecode and using the debugger. If you’re working in a Java environment I’d strongly encourage you to learn the IDE your developer’s use and how the debugger works there. Learning to navigate the codebase and understanding what live thread and stack dumps look like can really help you reason better about the application when troubleshooting.

Test-driven debugging

This is an expansion of the previous point - some language debuggers such as pdb or pry make it easy to add a breakpoint programatically within the test, or using the IDE or even just running the test in the debugger set the breakpoint and step into.

I’ve often used this as a way to improve my understanding of the code under test and the API before writing a new test. It allows you to drill deep.

Debugging production issues

At various points in my career I’ve had to dive in at quite a low level and figure out what is happening - with large and complex multithreaded applications this can be non-trivial. Often you really just want to get the stack trace of the program when the issue manifests.

You want to apply careful thought before ever doing this.

  • Come up with a hypothesis
  • Attempt to isolate problem
  • Run with debugger in isolated environment

file descriptor leaks within a large JVM application including JNI database client.

When facing these issues gdb can be very powerful - in addition to the example of running the program in gdb you can attach to processes or analyse core dumps.

Scripting your debugger

gdb supports various forms of automation and scripting - for example you can define helper functions. One reason I’ve used this in the path is when debugging the JVM - the JVM users various signals internally so we can to configure gdb to ignore these.

define java_signals
  handle SIGTSTP noprint nostop
  handle SIGSEGV noprint nostop
  handle SIGQUIT noprint nostop
  handle SIGABRT stop print nopass 
end

SIGQUIT gives us a JVM stacktrace so we can understand how where we are natively corresponds to where we are in our Java codebase. You can also define a gdb function to force that.

define jtrace
  call kill(getpid(), 3)
  cont
end

The times I’ve had to connect debugger to real live production tasks - this should always be done with caution - if you can ensure it’s not serving real user traffic either by is best as debuggers suspend execution which can impact user requests. You also may find that your monitoring or health checking may fail and kill the process under the debugger. Often a stacktrace is enough.

gdb supports attaching commands on a breakpoint so you can run some code so you could for example run the above jtrace function to force a Java stack trace when you hit some specific state and continue. If you’re absolutely stuck with no alternative but to run the debugger on a production serving process practice in a test environment first to learn how to use breakpoint commands to minimize the amount of time the program will be suspended.

In addition to gdb scripting modern gdb supports python to write richer extensions, pretty printers, etc. Two examples I’ve used as reference are gdb-heap which allows inspection of python’s memory usage and Go’s runtime-gdb.py which is an interesting way to learn some of the runtime internals by seeing how it looks up goroutines, etc.

Conclusion

So we’ve learned the basic debugger primitives of

  • Setting a breakpoint
  • Inspecting some state
  • Advancing execution by stepping over a call
  • Debugging further in by stepping into a call

We’ve taken a look at some actual real world tasks where debugger knowledge is useful.

Working with multithreaded, concurrent and complex applications with a debugger is more challenging than the simple walk through here. Like many tools in your portfolio it’s best to practice using your debuggers before you’re in a situation where that’s the only way you could get information.

Further Reading

December 5, 2014

Day 5 - How To Talk About Monitors, Tests, and Diagnostics

Written by: Yvonne Lam (@yvonnezlam)
Edited by: Jennifer Davis (@sigje)

Over time, I’ve accumulated a short list of statements about test-like things, a personal category that includes monitors and diagnostics as well as traditional tests, that when I hear them make my heart sink into my shoes:

  • “That isn’t a unit/integration/functional test!”
  • “Why do we need a check for that?”
  • “We need to run our functional tests against the production system all the time and alert when they fail.”

These remarks are strong signals that we lack shared context about what we are trying to accomplish with the tests/monitors/diagnostics in question. Ensuing conversation is likely to be difficult unless we can build some.

I think of the context for a test-like thing in terms of five questions:

  1. Where is it going to run?
  2. When is it going to run? (e.g. What events will cause it to run?)
  3. How often will it run?
  4. Who is going to consume the result? (Answers could be “another service”, “an application that generates time-series data”, “a person”, or a chain of any of the above.)
  5. What is that entity or collection of entities going to do with the result?

These questions compose a heuristic, so I don’t feel that all of them need to be answered precisely every time.

For example:

  • A traditional test runs against a non-production system when the system is built or deployed. It will generally run because of an event such as code change or deployment. It will likely run relatively infrequently, e.g. hours rather than minutes or seconds. A person may look at the result of the test with the goal of fixing a bug before it goes into production, or an automated system may look at the aggregate test results in order to decide on the flow of code through a continuous integration or deployment system.
  • A monitor runs against a production or production-like system on a schedule. Most monitors run frequently, as in on minute or sub-minute intervals, although some may trigger based on other events. The result may be consumed directly by an alerting system such as nagios, or by another system to produce time-series or other data. In the first case, a result may cause an alert to fire, resulting in an alarm going off and potentially a person being paged. In the second case, measured changes in the state of the collected data may cause an alert to fire and a person to be notified, or the collected data may be used for other kinds of analysis or presented in other forms. The eventual consumer of the data will be someone who wants to know the state of the running production system.
  • A diagnostic runs against a production or production-like system on demand. It will be run when there is reason to believe that something is wrong. The result will be used to fix the running system, through reconfiguration or code change/deploy. In sufficiently advanced systems, cheap diagnostics may be run all the time, in order to trigger automated healing of common system errors or conditions.

Note that intent matters: checking to see if, say, nginx is available and serving data may be a test, a monitor, or a diagnostic. footnote1

An immediate benefit of this approach is that it allows discussion of test-like things without first wading through a sea of vocabulary. You don’t have to argue about what the difference between an integration test and a functional test is, or struggle for words to explain why converting unit tests to production monitors is probably not going to be the best use of the time you have to spend on monitoring. For the sake of effectiveness, it’s good to converge on shared terms, but if people already have different vocabulary, then that convergence will be eventual, and you may need test-like things before then.

The main reason why I like this approach is that it allows me to be very specific about what the test-like thing under discussion is supposed to do. Being able to say,

  • “This test-like thing writes data to a database. Do we want it to run every time someone makes a code change, given that we need to set up the database before the test and clean it up afterward?” (probably not), or
  • “If we think something is broken, we need to check that data is getting from A to B. Is that something we want to check once every ten seconds when the system is otherwise healthy?” (it depends)
  • “Our functional tests for account setup create new accounts every time. Do we want to create a new account in production every minute?” (probably not if we can alert on failed attempts and/or unusually low counts of new accounts per unit time) footnote2

makes for a more useful conversation, in my opinion.

Specificity is especially useful when people ask for a type of test or monitor that is not consonant with the way the application or service component under examination has been written. For example, a common response to user-facing bugs that depend on system state and hence are not easy to reproduce is, “Write a functional test and run it as a monitoring check.” This is an excellent idea, since it allows one to track how often the problem happens, get closer to understanding what system state is relevant, etc. It also may not be practical. A test that replicates a user scenario is powerful in part because it exercises the entity being tested at a high level of abstraction; attempts to capture that test in code often results in test code that is fragile, requires more access to service or application internals than is readily available, or involves expensive setup and teardown to run at the frequency required. The more specific you can be about what such a monitoring check actually needs to do and what that requires, the better equipped you are to suggest alternatives, or, in the most extreme case, talk about what would need to change in the application in order for it to be monitored as desired.

Gratuitous piece of advice #1: It’s worth considering whether some of those health checks that run all the time could be replaced either with monitors that feed a system for collecting data for trend analysis or with diagnostics that a person could use to probe a running system when they suspect something is wrong. Gratuitous piece of advice #2: It depends on your service, but if it is data-intensive and/or relies heavily on caching to improve performance, using your functional tests as monitors may not get you the result that you want. Services whose primary purpose is data collection and storage are most likely to get use out of functional tests as monitors. For a service that does real-time or near-real-time data mining, I have found that data-dependent functional tests either are or become problematic as monitors.

December 4, 2014

Day 4 - Introduction to Kafka

Written by: Brandon Burton (@solarce)
Edited by: Scott Murphy (@ovsage)

There are many new distributed systems being released these days and it can be hard to keep up. Apache Kafka is one that seems to have recently seen a huge uptick in adoption as a message/event bus in many organizations, including my own work place, Lookout.com. Since we began to use it and I’ve been learning how to run it in production over the last few months, I wanted to provide an introduction to the operational side of Kafka. I won’t really get into using Kafka as part of your application but will provide some jumping off points to where to learn more about accomplishing that near the end of this post.

Kafka Mission Statement

So what is Apache Kafka?

According to the Project Site, Apache Kafka is …

Written in Scala and runs on the JVM.

publish-subscribe messaging rethought as a distributed commit log.

Fast

A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.

Scalable

Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers.

Durable

Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.

Distributed by Design

Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.

Breaking it down

That all sounds pretty good right? But what does all that mean?

First let’s review some basic messaging terminology:

  • Kafka maintains feeds of messages in categories called topics.
  • Topics are divided into partitions for scaling and redundancy.
  • We’ll call processes that publish messages to a Kafka topic producers.
  • We’ll call processes that subscribe to topics and process the feed of published messages consumers.
  • Kafka is run as a cluster comprised of one or more servers each of which is called a broker.

So, at a high level, producers send messages over the network to the Kafka cluster, the messages are stored in a topic, and the cluster serves them up to consumers.

Let’s dive into a little more detail on each of these facets of Kafka.

Brokers

A broker is a node in a Kafka cluster. Each broker acts as a leader for one or more topic partition, depending on the topic partition count settings. Each broker will also act as an in-sync replica (ISR) for additional topic partitions, depending on the replication factor of the topic. A broker recieves messages from one or more producters, on a per topic basis, and delivers messages to consumer of a topic’s partitions.

kafka cluster example

Topics

A topic is the abstraction that acts as a bucket for messages being published to a Kafka cluster. A topic is divided into one or more partitions, which are ordered log files on disk.

Partitions

Each partition is an ordered, immutable sequence of messages that is continually appended to — a commit log. The messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition. Each partition within a topic can have one or more in-sync replicas, to provide failover, based on a per topic replication factor setting. All writes and reads go throught the elected leader of a partition and a message is considered “committed” when all in sync replicas for that partition have applied it to their log. Only committed messages (in that all ISRs) are ever given out to the consumer(s).

partition example

Producers

Producers publish data to the topics of their choice. The producer is responsible for choosing which message to assign to which partition within the topic. This can be done in a round-robin fashion simply to balance load or it can be done according to some semantic partition function (say based on some key in the message).

Consumers

Consumers label themselves with a consumer group name, and each message published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.

If all the consumer instances have the same consumer group, then this works just like a traditional queue balancing load over the consumers.

If all the consumer instances have different consumer groups, then this works like publish-subscribe and all messages are broadcast to all consumers.

Zookeeper

One things that catches many people by suprise is that Kafka has a hard dependency on the Apache Zookeeper project, especially when coming from using another system like RabbitMQ or Redis as your message/event bus.

What is it?

ZooKeeper is a distributed system which provides a strongly consistent interface to a hierarchical file system like structure. It provides an event-driven model in which clients can watch for changes to specific nodes, called znodes. An example of such a change could be a new child being added to an existing znode. ZooKeeper achieves high availability by running multiple ZooKeeper servers, called an ensemble, with each server holding an in-memory copy of the distributed file system to service client read requests. Each server also holds a persistent copy on disk. A zookeeper ensemble has strict rules about how it achieves a quorum in order to achieve its strong consistency guarantees.

What does Kafka use it for?

Zookeeper is used in a Kafka cluster to coordinate which brokers are members of a cluster and to coordinate leader election for in-sync replication of topic partitions. Since Zookeeper was already built as a strongly consistent distributed system for storing this kind of state, it made sense for the Kafka project to utilize Zookeeper instead of attempting to rebuild these features inside the Kafka codebase. Zookeeper’s design though is not meant for write-heavy workloads. This means that while Kafka relies on Zookeeper for broker discovery and ISR leader election, it also does some things entirely within the Kafka cluster, such as ensuring that a write is fully committed to all ISRs before acknowledging the write to a producer.

Deploying Kafka and Zookeeper

So now that we’ve learned a little about Kafka and Zookeeper, let’s review some key points on deploying it.

  • Zookeeper ensembles should be deployed in odd number sizes in order for its quorum to be maintained. The minimum recommended Zookeeper ensemble size is 3 nodes for staging environments and 5 nodes for production. This provides fault tolerance and the ability to perform maintainence while still maintaining N+1. Due to the nature of Zookeeper’s design, going over 5 nodes can start to cause increases in latency. You can read more on that on http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview.
  • The Kafka cluster size will vary depending on your workload, your desired partition count and replication factor settings. We’re using a default replication factor of 3, which means we use 3 node clusters for staging and 5 node clusters for production.
  • The recommended approach is to deploy a local Kafka cluster in each datacenter and have machines in each location interact only with their local cluster. For applications that need a global view of all data you should use the mirror maker tool to provide clusters which have aggregate data mirrored from all datacenters. These aggregator clusters are used for reads by applications that require the global view.
  • Monitoring of Kafka is done primarily through JMX and there is a great overview of the key MBeans to monitor and what they mean.

Beyond that, you should go straight to the Kafka Operations documentation as it’s very well written and you’ll want to read it all in-depth before tackling a Kafka deployment.

Further Reading

So where to go from here?

To learn more about Kafka in general, a few things I’d recommend, in no specific order are:

Operating Kafka and Zookeeper

Using Kafka in Your Application

December 3, 2014

Day 3 - So Server, tell me about yourself. An introduction to facter, osquery and sysdig

Written by: Gareth Rushgrove (@garethr)
Edited by: Hugh Brown (@saintaardvark)

Introduction

Linux and Unix have always had powerful, low level tools capable of telling you exactly what your computer system is doing (strace, DTrace, systemtap, top, ps). But these tools often have complex user interfaces, and platform differences, that mean not everyone has the time to master them. This article is all about several new tools that aim to not just be powerful debugging tools, but to provide a pleasant user interface too.

Facter is a simple inventory application providing a single, cross-platform interface to a range of structured data about your system. Everything; from network interfaces to available hardware and operating system version is available.

osquery is a new open source tool from Facebook that exposes low level details of your system via a familiar SQL interface. Want to query for processes listening on a given network interface? Or for services that launch at startup? This is the tool for you.

Sysdig is another open source tool for system level exploration and tracing that aims aiming at being both powerful and easy to use. Sysdig focuses on tools to help answer real-time issues.

Installation

I’m running all of the following on an Ubuntu 14.04 virtual machine, but you should be able to find the installation commands for your favourite distribution too. As for supporting other operating systems: Facter also runs on Windows and OS X; osquery also runs on OS X; and Sysdig is Linux only.

Facter

Facter has been around for a while (it’s a core part of Puppet), and is included in lots of distribution repositories already. However, for this walkthrough, we’re going to use the preview version of facter.

First let’s install the official Puppet labs repositories:

wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
sudo dpkg -i puppetlabs-release-trusty.deb

Next let’s install the nightly build repository for facter. Note that the repository and package are called cfacter to allow it to be installed alongside the stable version of facter.

cd /etc/apt/sources.list.d
sudo wget http://nightlies.puppetlabs.com/cfacter-latest/repo_configs/deb/pl-cfacter-latest-trusty.list

For the curious, or those wanting to use a different operating system. feel free to read -up on the nightly repositories.

osquery

osquery is quite new, and packages aren’t available just yet – so we’ll need to compile from source. First let’s download the latest release:

wget https://github.com/facebook/osquery/archive/1.1.0.tar.gz
tar -zxvf 1.1.0.tar.gz
cd osquery-1.1.0

And then we’ll install its dependencies and compile the osquery tools. This will take a little while but I promise it will be worth it.

sudo make deps
sudo make
sudo make install

For the full installation instructions see the osquery wiki.

Note that if you want to use osquery for anything more than a quick demo you could create your own package using the makefile.

make package

The resulting system package (Ubuntu or Centos at the moment) can then be used to install the binaries without needing to compile everywhere.

Sysdig

Sysdig handily provide a one-line installer which detects your operating system and installs the relevant packages:

curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

If you would rather do that manually then full installation instructions are available.

Usage

Facter

Facter is the most straightforward of the three tools we’re taking a look at. When run, it simply outputs structured information about the host, collected from various other tools or the operating system itself. This can be hugely useful if you’re on a machine and want to know everything quickly – but it’s also useful if you’re using an unfamiliar operating system, as it provides a single way of accessing lots of information quickly.

The quickest way of understanding this is just to run it:

cfacter | head -n 20

Feel free to leave out the pipe to head if you’re running locally.) The output is over 100 lines long, and looks something like this:

cfacterversion => 0.2.0
disks => {
  sda => {
    model => "VBOX HARDDISK",
    size => "40.00 GiB",
    size_bytes => 42949672960,
    vendor => "ATA"
  }
}
dmi => {
  bios => {
    release_date => "12/01/2006",
    vendor => "innotek GmbH",
    version => "VirtualBox"
  },
  board => {
    manufacturer => "Oracle Corporation",
    product => "VirtualBox"
  },
  chassis => {

You can see that I’m running this on a VirtualBox virtual machine with a 40GB hard drive.

Facter supports other output formats too, including JSON and YAML. For instance you can run:

cfacter -y | head

And you’ll receive YAML:

cfacterversion: 0.2.0
disks:
  sda:
    model: VBOX HARDDISK
    size: 40.00 GiB
    size_bytes: 42949672960
    vendor: ATA
dmi:
  bios:
    release_date: 12/01/2006

Facter also supports returning just a single value, so if you know the name of the fact you want to check you can simply ask for that. For instance:

cfacter ruby.version
1.9.3
cfacter os.distro.codename
trusty

As well as the large number of facts provided out-of-the-box on a range of operating systems, facter also allows for writing your own facts. A very simple example might be exposing the version of python to facter. First write a script that outputs a simple key=value pair. Save the following as /etc/facter/facts.d/python-version.sh.

#!/bin/bash
var=$(python --version 2>&1)
echo "python_version=$var"

Now we can ask for the python_version fact (the name of our key in the script above) like so:

cfacter --external-dir /etc/facter/facts.d/ python_version
Python 2.7.6

Facter has a number of different ways of extending it and any custom facts from previous versions of Facter should work with the new implementation.

osquery

osquery services a similar purpose to Facter, providing a universal interface for information on a machine. osquery presents information about the system as tables, which can be queried via SQL. The information being queried tends to return a dynamic list of results – for instance, the users present on a machine or the host entries in the local hosts file. Again, here’s a quick example:

echo "SELECT * FROM etc_hosts;" | osqueryi

This will output something like the following:

+-----------+----------------------------+
| address   | hostnames                  |
+-----------+----------------------------+
| 127.0.0.1 | localhost                  |
| ::1       | ip6-localhost ip6-loopback |
| fe00::0   | ip6-localnet               |
| ff00::0   | ip6-mcastprefix            |
| ff02::1   | ip6-allnodes               |
| ff02::2   | ip6-allrouters             |
| ff02::3   | ip6-allhosts               |
+-----------+----------------------------+

Let’s change the host entries on our machine and rerun the query:

echo "127.0.0.1 testingosquery" | sudo tee -a /etc/hosts
echo "SELECT * FROM etc_hosts;" | osqueryi

Now you’ll see something like:

+-----------+----------------------------+
| address   | hostnames                  |
+-----------+----------------------------+
| 127.0.0.1 | localhost                  |
| ::1       | ip6-localhost ip6-loopback |
| fe00::0   | ip6-localnet               |
| ff00::0   | ip6-mcastprefix            |
| ff02::1   | ip6-allnodes               |
| ff02::2   | ip6-allrouters             |
| ff02::3   | ip6-allhosts               |
| 127.0.0.1 | testingosquery             |
+-----------+----------------------------+

The above examples use the osqueryi tool which can take a query on stdin and return the results. You can also run osqueryi on it’s own and open an osquery SQL shell.

osqueryi

With the shell open, lets build a more complex query by joining together two tables.

osquery> SELECT
    ...> u.username,
    ...> g.groupname
    ...> FROM users as u
    ...> JOIN groups as g ON u.gid = g.gid
    ...> LIMIT 10;

This should produce something like the following:

+----------+-----------+
| username | groupname |
+----------+-----------+
| root     | root      |
| daemon   | daemon    |
| bin      | bin       |
| sys      | sys       |
| sync     | nogroup   |
| games    | games     |
| man      | man       |
| lp       | lp        |
| mail     | mail      |
| news     | news      |
+----------+-----------+

osquery supports a large and growing number of tables – everything from arp_cache and bash_history, to crontab records and kernel_modules. It’s also possible to write your own tables if you’re happy getting your hands into the code.

osquery also supports a long-running daemon process called osqueryd; this allows for scheduling queries for execution across your infrastructure, aggregating the results over time and generating logs of any changes in state.

Sysdig

Whereas Facter and osquery are predominantly about querying infrequently changing information, Sysdig is much more suited to working with real-time data streams – for example, network or file I/O, or tracking errors in running processes.

Here’s a few examples. First let’s watch for any operations that open the /etc/hosts file:

sudo sysdig evt.type=open and fd.name contains /etc/hosts

Now in another tab or ssh session, open the /etc/hosts file with vim or other editor of choice:

vim /etc/hosts

This should output something like the following:

3145 12:05:06.477169760 1 vim (3835) < open fd=3(<f>/etc/hosts) name=/etc/hosts flags=1(O_RDONLY) mode=0

Here we can see that vim made an open syscall to the /etc/hosts files.

Let’s do something a bit more practical: we’ll look for any I/O calls that have a latency greater than 1ms. This would be useful if you were tracking down certain kinds of performance issues:

sudo sysdig -c fileslower 1

fileslower is what’s called a Chisel, you can find out more on the Chisel user guide or read this tutorial about how to write your own.

In another tab or session let’s run a command that should trigger a bit of I/O. We’ll use these packages in the next example too.

sudo apt-get install nginx apache2-utils -y

On the virtual machine I ran this on this resulted in the following.

2014-11-28 12:10:02.821 dpkg         read           1 /var/lib/dpkg/info/linux-headers-3.13.0-34.list
2014-11-28 12:10:04.517 mandb        read           1 /usr/share/man/man5/resolv.conf.5.gz
2014-11-28 12:10:06.323 apt-get      read          15 /var/cache/apt/srcpkgcache.bin

The output here is showing any files where the I/O latency was greater than 1ms. Each line shows the binary (apt-get, dpkg, etc.), the action (read in this case) and the latency (1ms or 15ms). If you were using sysdig to debug a real performance problem this kind of information should be much more useful.

I mentioned above that we’d make use of the nginx and apache-utils packages for our next example. Let’s watch all the events related to requests served by nginx in real time.

sudo sysdig -A -c echo_fds proc.name=nginx

And again in another tab or session, let’s run apache bench to generate some traffic against our local nginx web server.

ab -n 5 -c 1 http://localhost/

This should output something like the following:

------ Read 77B from 127.0.0.1:58089->127.0.0.1:80

GET / HTTP/1.0
Host: localhost
User-Agent: ApacheBench/2.3
Accept: */*

------ Write 241B to 127.0.0.1:58089->127.0.0.1:80

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 28 Nov 2014 12:17:02 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 04 Mar 2014 11:46:45 GMT
Connection: close
ETag: \"5315bd25-264\"
Accept-Ranges: bytes

------ Write 90B to /var/log/nginx/access.log

127.0.0.1 - - [28/Nov/2014:12:17:02 +0000] \"GET / HTTP/1.0\" 200 612 \"-\" \"ApacheBench/2.3\"

Note that we’re seeing the request, the response and the log lines being written - all from the same command and all in real time. Imagine how useful that would be when debugging a production web server.

The folks behind Sysdig provide lots of examples which gives you an idea of all the possibilities: from watching the behaviour of particular users, to tracking busy processes, to recording users of a specific application.

Conclusions

Hopefully these quick examples have given you an insight into three useful tools and into why you might want them around when you have a problem. All three of these tools present lots of opportunities for integration with your monitoring or configuration management framework.

December 2, 2014

Day 2 - Running Applications at Scale with Kubernetes

Written by: Kelsey Hightower (@kelseyhightower)
Edited by: Tom Purl (@tompurl)

In this post we’ll take a look at Linux containers, built using the Docker image format, as an application packaging and distribution mechanism. This means utilizing containers as an alternative to RPMs, Debs, and tarballs. Then we’ll turn our attention to the workflow provided by Kubernetes for managing applications.

Normally when we talk about scale we tend to think about providing more compute resources to run applications, but today I’m talking about the ability to truly separate concerns between building applications and running them in production. Today I want to talk about scaling how we ship applications.

Containers for application packaging

The key to automation is to simplify things before you automate them. I’m a big fan of self-contained binaries like the ones produced by the Go compiler. It’s hard to argue against the convenience of distributing a single artifact vs the “deploy and pray” method offered by many of today’s packaging solutions.

Each language community has attempted to solve the packing problem in various ways specific to their platform needs. As a result we have gems, bundler, pip, virtualenv, npm, and the list goes on and on. The drawback to those solutions? They don’t allow you to express the OS dependencies required by many of the applications we build and ship today. With Docker containers we effectively gain the ability to build self-contained binaries for any application stack.

To be clear, Docker’s image format does not solve the packaging problem, it just moves it around a bit. The end results are pretty good. We can now rely on a single artifact to universally represent any application. The Docker container has become the new lego brick for building larger systems.

Now I can focus on more interesting problems.

The Datacenter is the Computer

With containers I can easily ship applications between machines and start to think of a cluster of machines as a single computer. Each machine acts as another CPU with the ability to execute applications and runs an operating system, but the goal is not to interact with the local OS directly. Instead we want to treat the local OS as firmware for the underlying hardware resources.

The only thing missing now is a scheduler.

The Linux kernel does a fantastic job of scheduling applications on a single system. Chances are if we run multiple applications on a single system the kernel will attempt to use as many CPU cores as possible to ensure that our applications run in parallel.

When it comes to a cluster of machines the job of scheduling applications becomes an exercise for the sysadmin. Today for many organizations scheduling is handled by the fine men and women running Ops. Unfortunately, human schedulers require humans to keep track of where applications can possibly run. Sometimes this means using spreadsheets or a configuration management tool. Either way these tools don’t offer robust scheduling that can react to real time events. This is where Kubernetes fits in.

If the datacenter is the computer then Kubernetes would be it’s operating system.

Application scheduling with Kubernetes

Kubernetes is a declarative system for scheduling Docker containers. First we need to get on the same page regarding what I mean by declarative. In Kubernetes we don’t write infrastructure as code. I know this goes against what many have been taught with regards to thinking about infrastructure, but let me try to explain.

Kubernetes introduces the concept of a Pod, a collection of Linux containers, which represents a single application. Kubernetes also provides a specification for declaring to the cluster how Pods should be deployed. As a result we only need to express our applications like this:

id: helloController
apiVersion: v1beta1
kind: ReplicationController
desiredState:
  replicas: 2
  # replicaSelector identifies the set of Pods that this
  # replicaController is responsible for managing
  replicaSelector:
    name: hello
  # podTemplate defines the 'cookie cutter' used for creating
  # new pods when necessary
  podTemplate:
    desiredState:
      manifest:
        version: v1beta1
        id: hello
        containers:
          - name: hello
            image: quay.io/kelseyhightower/hello:1.0.0
            ports:
              - containerPort: 80
    # Important: these labels need to match the selector above
    # The api server enforces this constraint.
    labels:
      name: hello

The above configuration represents the hello application as a Kubernetes pod composed of a single container, quay.io/kelseyhightower/hello:1.0.0, listening on port 80.

Once we have written this configuration and feed it to the Kubernetes API the various components that make up Kubernetes will take the following steps:

  • Locate nodes capable of running the Pod.
  • Chosen nodes will download and run the quay.io/kelseyhightower/hello:1.0.0 container.
  • A process will monitor the state of the Pods to ensure 2 copies are running at all times.

At this point we have declared to the system that we require 2 instances of the hello pod to be running at all times. If instead we required 1000 instances of the hello app, we could adjust the configuration, then add more machines to the cluster. Once the additional compute resources show up the Kubernetes scheduler would deploy our pods to them until the desired number of pods were running.

If one of the nodes running a pod were to go offline, then a new pod will be created and scheduled to the next eligible node in the cluster. Kubernetes enforces our definition at all times. This is what makes Kubernetes a declarative system.

It’s all about workflow

The dream work flow is that code gets checked in, some tests would run, and things would start magically running in production. For those that have tried pulling this off, you know that’s not an easy thing to do.

Kubernetes offers a step in the right direction. Code gets checked in and then containers are built and pushed to registries. Once this happens the team or person in charge for that service can simply update the Kubernetes configuration to pick up the new container. That workflow looks like this:

kubecfg --image "quay.io/kelseyhightower/hello:2.0.0" rollingupdate helloController

So far I’ve only provided a high level view of Kubernetes, but it only the beginning. Kubernetes provides integration with DNS, load balancers, and even centralized logging. The ideas behind Kubernetes are solid, and anything missing can be easily added to build the right system for your needs.

What’s in it for Sysadmins?

Tools like Kubernetes feel like the system I’ve always tried to build. Some see Kubernetes as a PaaS like Heroku, but that’s far from the case. Kubernetes is a pluggable system that provides a platform for managing application containers at scale.

I just want to be clear, Kubernetes is not some magical system that paints rainbows and trains ponies. It’s a well defined system that captures the experience of fellow sysadmins into an open source project we can all reuse and contribute to. The idea is that Ops would keep Kubernetes and the underlying machines up and running to provide a complete platform for running any application or job required by the team. I like to think of Kubernetes as Ops with an API.

If you would like to learn more about Kubernetes checkout the official website, or take it for a spin with the Intro to Kubernetes Tutorial

December 1, 2014

Day 1 - Docker in Production: Reality, Not Hype

Written by: Bridget Kromhout (@bridgetkromhout)
Edited by: Christopher Webber (@cwebber)

Why Docker?

When I started talking with DramaFever in summer 2014 about joining their ops team, one of many appealing factors was that they’d already been running Docker in production since about October 2013 (well before it even went 1.0). Cutting (maybe bleeding) edge? Sounds fun!

But even before I joined and we were acquired by SoftBank (unrelated events! I am not an acquisition magnet, even if both startups I worked at in 2014 were acquired), DramaFever was already a successful startup, and important technology stack decisions are not made by running a Markov text generator against the front page of Hacker News.

So, why Docker? Simply put, it makes development more consistent and deployment more repeatable. Because developers are developing locally all using the same containers, integration is much easier when their code moves on to their EC2-based personal dev environment, the shared dev environment, QA, staging, and production. Because a production instance is serving code from a container, every new autoscaled instance that has any code at all is going to have the correct code.

As renowned infosec professional Taylor Swift says, “haters gonna hate”. And I’ve been guilty of “get off my lawn” snark about the recent hype, pointing out that containerization isn’t exactly new. We’ve had FreeBSD chroot jails and Solaris Zones (pour one out for Sun Microsystems) for ages. But the genius of Docker is that it provides just enough training wheels for LXC that everyone can use it (for rapidly increasing values of everyone).

Our Own Private Registry

We’re using a local copy of the registry backed by an S3 bucket accessible to those with developer IAM credentials. (If you don’t use AWS, that just means it uses a shared storage location that our devs can access without needing production keys.) Apparently people usually go with a centralized private registry; instead, we traded a SPOF for S3’s eventual consistency. We start the local registry on a host via upstart, and there are a few configuration items of interest:

# this goes in /etc/default/docker to control docker's upstart config
DOCKER_OPTS="--graph=/mnt/docker --insecure-registry=reg.example.com:5000"

Since on some instances we are pulling down multiple Docker images that can be hundreds of megabytes in size, and running or stopped containers also take up room on disk, we use --graph=/mnt/docker to set the root of the docker runtime to the ephemeral disk instead of to the default /var/lib/docker.

With Docker 1.3’s improved security requiring TLS, this means we need to allow our localhost-aliased non-TLS registry.

The docker registry upstart job (used on all EC2 instances) runs these commands:

docker pull public_registry_image
docker run -p 5000:5000 --name registry \
-v /etc/docker-reg:/registry-conf \
-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml \
public_registry_image

That second line may require some explanation:

# this is the local port we'll run the registry on
docker run -p 5000:5000 \  

# giving the container a name makes it easier to identify
--name registry \  

# we're mounting in the directory holding a config file that specifies
AWS credentials, etc. Unlike mount(1), this creates the directory it's
mounting to.
-v /etc/docker-reg:/registry-conf \  

# defining the config file location
-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml

# the publicly-registered image we're launching this local registry from
public_registry_image

To run locally, we pull the image and then run like this (with DFHOME being where we have the source code checked out):

docker run -d -p 5000:5000 --name docker-reg -v ${DFHOME}:${DFHOME} -e 
DOCKER_REGISTRY_CONFIG=${DFHOME}/config/docker-registry/config.yml
public_registry_image

docker build; docker push

Weekly Jenkins jobs build a base container for the main django app and another that mimics our RDS environment with a containerized, all-data-fixtures-loaded MySQL.

We do trunk-based development with developers submitting pull requests. After being peer-reviewed and merged to master, the new code is available for Jenkins to poll GitHub and build. If all tests pass, then it’s time for exciting post-build action! (What? If you’ve gotten this far in a post about container strategy, then you probably agree with me that this stuff is exciting.)

While all our Go microservices are built essentially the same way, let’s focus on the main django app. Its Dockerfile starts from the weekly base build, as that speeds things up a bit:

FROM our-local-repo-alias:5000/www-base

We keep a number of Dockerfiles around, and in this case, since we have both a base build and a master build for www, we have multiple Dockerfiles in this github repository. Since it’s not possible to pass a file to docker build, it’s necessary to rename the file:

mv 'Dockerfile-www' Dockerfile; sudo docker build -t="67cd893" .

Jenkins builds the new layers for the www master image, tags it with the git SHA, then tests it. Only if it passes the tests do we retag it as dev and then push it to our private docker repository.

sudo docker push our-local-repo-alias:5000/www:'dev'

When we’re ready to cut a release, we build the www-QA job from the release branch. After testing, that same container is re-tagged for staging, then production, and new autoscaling instances will pick it up (giving us the flexibility to do blue/green deploys, which we’re just starting to explore).

Docker in a Mac-using Dev World

Before summer 2014, we were using Vagrant for local development. Building a new image with a local chef-solo provisioner took 17 minutes to install everything, and the local environment diverged enough from the production environment to be annoying. Moving all development into Docker containers proved very effective, especially as we worked through some of the inevitable gotchas and corner cases.

For the local developer environment, we’re using boot2docker, and we’re just about to move back to mainline from a fork that Tim Gross, our head of operations, wrote to get around VirtualBox shared folder mount issues present in previous versions of boot2docker.

One issue we’ve noticed using boot2docker on Mac laptops is that when they wake from sleep, the clock in the VM can be skewed. This means the local registry doesn’t work, since it relies on S3 and S3 expects a correct clock.

$ boot2docker ssh sudo date -u
Mon Nov 24 16:09:02 UTC 2014

$ date -u
Tue Nov 25 01:43:49 UTC 2014

$ docker pull our-local-repo-alias:5000/mysql
Pulling repository our-local-repo-alias:5000/mysql
2014/11/24 19:44:31 HTTP code: 500

Ry4an Brase, our head of back-end development, came up with this delightful incantation:

$ boot2docker ssh sudo date --set \"$(env TZ=UTC date '+%F %H:%M:%S')\"

Adding that to the utils sourced by all our various wrapper scripts (so that devs don’t need to remember a lot of docker syntax to go about their daily lives) seemed like a better alternative than having slackbot deliver it as a reply to all local registry questions.

Containerizing Front-End Dev

I created a container for front-end development which allows us to replicate a front-end environment on Jenkins, using angular, npm, grunt, and bower; you know, the sort of mysterious tools that are inordinately fond of $CWD and interactive prompts.

There are a number of Dockerfiles out there for this; here’s what I found helpful to know (for values of “know” that include “asking Ryan Provost, our head of front-end development” and “mashing buttons until it works”).

Although it defies all logic, node is already old enough to have legacy something. (Insert rant about you kids needing to get off my lawn with your skinny jeans and your fixies.)

RUN apt-get install -y nodejs nodejs-legacy npm

You need a global install of these three; they can’t come from your package.json:

RUN npm install -g grunt-cli@0.1.13
RUN npm install -g bower@1.3.8
RUN npm install -g phantomjs@1.9.7-14

And bower doesn’t want to be installed as root, and sometimes will ask questions that expect an interactive answer:

ADD bower.json /var/www/dependencies/bower.json
RUN cd /var/www/dependencies && bower install --allow-root 
--config.interactive=false --force

The nice thing about having this container is that it allows someone without all the right versions of the front-end tools installed to try out running such parts of the site locally, and it also allows a more replicable deployment of something that will definitely be the same between all the environments as opposed to the “it works on their laptop” fun we all know and love.

Getting the Logs Out

A certain Docker Orthodoxy treats a container as entirely apart from the host instance. Since we aren’t using containers for isolation, we approach this a little differently. Tim blogged about Docker logs when DramaFever first started using Docker.

On EC2 we want to ship logs to our ELK stack, so we mount in a filesystem from the host container:

-v /var/log/containers:/var/log

On local developer machines we want to be able to use a container for active development, editing code locally and running it in the containerized environment. We use the -v flag to mount the developer’s checked-out code into the container, effectively replacing that directory as-shipped:

-v ${DFHOME}/www:/var/www

We still want logs, too, so we expose those for the dev here:

-v ${DFHOME}/www/run:/var/log

Totally Weird Bugs for $1,000, Alex

On an instance where the docker runtime root disk filled up, the container images became corrupt. Even after a reboot, they started yielding inconsistent containers whose behavior would vary over time. For example, a running container (invoked with /bin/bash) would have the ls command, and then a few minutes later, it would not. Eventually, a docker run would lead to errors like these:

Error response from daemon: Unknown filesystem type on /dev/mapper/
docker-202:16-692241-81e4db1aaf5ea5ec70c2ef8542238e8877bbdb4b0
7b253f67b888e713a738dc2-init

Error response from daemon: mkdir /var/lib/docker/devicemapper/mnt/
9db80f229fdf9ebb75ed22d10443c90003741a6770f81db62 f86df881cfb12ae-init/
rootfs: input/output error

It’s likely that we’re seeing one of the devicemapper bugs that seem to plague docker. Replacing the local volume in question was a reasonable workaround.

About Those Race Conditions

A much more prevalent (and annoying) place we’ve run into docker race conditions is in the Jenkins builds. We’d been seeing builds fail with messages like this:

Removing intermediate container 4755dce8cfcc
Step 5 : ADD /example/file /example/file
2014/11/18 18:46:59 Error getting container init rootfs 
a226d3503180de091fde2a410e2b037fde94237dd2171d49a866d43ff03e724c from 
driver devicemapper: Error mounting '/dev/mapper/docker-9:127-14024705-
a226d3503180de091fde2a410e2b037fde94237dd2171d49a866d43ff03e724c-init'
on '/var/lib/docker/devicemapper/mnt/
a226d3503180de091fde2a410e2b037fde94237dd2171d49a866d43ff03e724c-
init': no such file or directory

I added the Naginator plugin so it would retry failed jobs if they’d failed with the most common strings we’d see:

(Cannot destroy container|Error getting container init rootfs)

While that’s an acceptable workaround, I still plan to change what gets reported to Slack, since it’s annoying to have to click on the broken build to find out if it’s just Docker again.

Cron Zombies

A few weeks ago, a developer noticed an unwelcome new message in interactive use on QA:

Error: Cannot start container appname: iptables failed: iptables -t 
nat -A DOCKER -p tcp -d 0/0 --dport 8500 ! -i docker0 -j DNAT --to-
destination 172.17.0.7:8500:  (fork/exec /sbin/iptables: cannot
allocate memory)

On specific instances (such as QA) that aren’t part of the production autoscaling groups, we run cron jobs that invoke a container and give it arguments. A look at the process table showed that multiple days of docker run commands started by cron and the python processes they’d spawned were still running. docker ps disagreed, though; the containers weren’t running anymore, so they weren’t getting cleaned up by these cron jobs:

# remove stopped containers
@daily docker rm `sudo docker ps -aq`
# remove images tagged "none" 
@daily docker rmi `sudo docker images | grep none | awk -F' +' 
'{print $3}'`

At the time, we were starting the cron containers with docker run -i -a stdout -a stderr.

Changing the container-invoking cron jobs to instead use docker run -it cleared it up. It appears a controlling tty was necessary for them to successfully signal their child processes to exit.

Containerize All the Things?

We’re actually running just about everything in containers currently - including the more static bits of our infrastructure. Do Sentry, Jenkins, Graphite, and the ELK stack actually benefit from being in containers? Possibly not; at the time of containerizing all the things, it was the closest thing we had to a configuration management system.

But while it’s excellent for releasing software, it’s a giant hassle sifting through all the changes in a monolithic “config” repo to figure out why the graphite container no longer builds to a working state. Now that we’re using Chef and Packer to drive our AMI creation, we’ll likely move to using Chef cookbooks to manage our next iteration on those infrastructure pieces.

Just because it’s possible to run everything inside a container doesn’t mean it’s useful. While we are no longer using containers as our main method of capturing all configuration, we continue to see great value in using them for consistency throughout development and repeatability of deployments.

Not (Just) Hype

The core of devops is empathy, and it’s important to remember there are people behind all the software we’re discussing. Nathan LeClaire of Docker took to Twitter recently, talking about how it feels to have your project called “marketing BS”. (Let’s pause for a moment while I feel guilty about everything I’ve ever said about MongoDB. I don’t think I ever called it marketing BS, but I’ve definitely made “web scale” jokes.)

Given the recent announcements out of AWS re:Invent about EC2 Container Service, it’s safe to say that containers are about as mainstream as they’re going to get. Do I think ECS is going to be ready for prime time immediately? Anyone who read my sysadvent post from last year about HBase on EMR (Amazon’s training wheels for Hadoop) is saying “lolnope” right about now.

But containers are definitely not just for the Googles of the world anymore, and they’re increasingly no longer just for those of us who are willing to chase devicemapper bugs down a rabbit-hole into GitHub issue land. Docker is the real deal, it works in production, and if you’d like to go stream some dramas powered by it from our site or native apps, you can do that today. (If you’d like to read more about containers, stay tuned for an exciting post tomorrow…)

Sound Like Fun?

If this sounds like exactly the sort of fun you enjoy having at work, we’re hiring ops and dev folks at DramaFever. We’re remote-friendly with NYC and Philly offices. You can read more about our positions on the DramaFever careers page or contact me via email or on twitter. I’d love to talk with you!