Day 19 - Troubleshooting Docker and Kubernetes

Written by: Jorge Salamero (@bencerillo)
Edited by: Brian O'Rourke (@borourke)

Container orchestration platforms like Kubernetes, DC/OS Mesos or Docker Swarm help towards making your experience like riding an unicorn over a rainbow, but don’t help much with troubleshooting containers:

• They are isolated, there is a barrier between you and the process you want to monitor and traditional troubleshooting tools run on the host doesn’t understand containers, namespaces and orchestration platforms.
• They bring a minimal runtime, just the service and its dependencies without all the troubleshooting tools, think of troubleshooting with just busybox!
• They are scheduled across your cluster… containers move, scale up and down. Are highly volatile, appearing and disappearing as the process ends, gone.
• And talk to each other through new virtual network layers.

Today we will demonstrate through a real use case how to do troubleshooting in Kubernetes. The scenario will use is a simple Kubernetes service with 3 Nginx pods and a client with curl. In the previous link you will find the backend.yaml file we will use for this scenario.

If you are new to Kubernetes services, we explained how to deploy this service and learned how it works in Understanding how Kubernetes DNS Services work.To bring up the setup, will run:

$ kubectl create namespace critical-appnamespace “critical-app” created$ kubectl create -f backend.yamlservice “backend” createddeployment “backend” created

And then will spawn a client to load our backend service:

$ kubectl run -it –image=tutum/curl client –namespace critical-app –restart=Never

Part1: Network troubleshooting Kubernetes services

From our client container we could simply run a test by doing root@client:/# curl backend to see how our Kubernetes service works. But we don’t want to leave things loose and we thought that using fully qualified domain names is a good idea. If we go and check Kubernetes documentation it says that every service gets this default DNS entry: my-svc.my-namespace.svc.cluster.local. So let’s instead use the full domain name.Let’s go back to the curl client container shell and run: root@client:/# curl backend.critical-app.svc.cluster.local. But this time curl hangs for 10 seconds and then correctly returns the expected website! As a distributed systems engineer, this is one of the worst things that can happen: you want something to fail or succeed straight away, not a wait of 10 seconds.To troubleshoot what’s going on, we will use sysdig. Sysdig is an open source linux visibility tool that offers native visibility into containers, including Docker, Kubernetes, DC/OS, and Mesos just to name a few. Combining the functionality of htop, tcpdump, strace, lsof, netstat, etc in one open source tool, Sysdig gives you all of the system calls and application data in the context of your Kubernetes infrastructure. Monitoring Kubernetes with Sysdig is a good introduction to using the tool with Kubernetes.

To analyze what is going on, we will ask sysdig to dump all the information into a capture file:$ sudo sysdig -k http://127.0.0.1:8080 -s8192 -zw capture.scap

I’ll quickly explain each parameter here:

-k http://localhost:8080 connects to Kubernetes API

-s8192 enlarges the IO buffers, as we need to show full content, otherwise gets cut off by default

-zw capture.scap compresses and dumps into a file all system calls and metadataIn parallel, we’ll reproduce this hairy issue again running the curl command: # curl backend.critical-app.svc.cluster.local. This ensures that we have all the appropriate data in the file we captured above to reproduce the scenario and troubleshoot the issue.Once curl returns, we can Ctrl+C sysdig to stop the capture, and we will have a ~10s capture file of everything that happened in our Kubernetes host. We can now start troubleshooting the issue either in the cluster or out of band, basically anywhere we copy the file with sysdig installed.$ sysdig -r capture.scap -pk -NA “fd.type in (ipv4, ipv6) and (k8s.ns.name=critical-app or proc.name=skydns)” | less

Let me explain each parameter here as well:

-r capture.scap reads from a capture file

-pk prints Kubernetes fields in stdout

-NA shows ASCII output

And the filter between double quotes. Sysdig is able to understand Kubernetes semantics so we can filter out traffic on sockets IPv4 or IPv6, coming from any container in the namespace critical-app or from any process named skydns. We included proc.name=skydns because this is the internal Kubernetes DNS resolver and runs outside our namespace, as part of the Kubernetes infrastructure.

Sysdig also has an interactive ncurses interface htop alike

In order to follow along with this troubleshooting example, you can download the capture file capture.scap and explore it yourself with sysdig.Immediately we see how curl tries to resolve the domain name but on the DNS query payload we have something odd (10049): backend.critical-app.svc.cluster.local.critical-app.svc.cluster.local. Seems like for some reason curl didn’t understand I gave it a fully qualified domain name already and decided to append a search domain to it.

[…]

10030 16:41:39.536689965 0 client (b3a718d8b339) curl (22370:13) < socket fd=3(<4>) 10031 16:41:39.536694724 0 client (b3a718d8b339) curl (22370:13) > connect fd=3(<4>) 10032 16:41:39.536703160 0 client (b3a718d8b339) curl (22370:13) < connect res=0 tuple=172.17.0.7:46162->10.0.2.15:53 10048 16:41:39.536831645 1 <NA> (36ae6d09d26e) skydns (17280:11) > recvmsg fd=6(<3t>:::53) 10049 16:41:39.536834352 1 <NA> (36ae6d09d26e) skydns (17280:11) < recvmsg res=87 size=87 data=backendcritical-appsvcclusterlocalcritical-appsvcclusterlocal tuple=::ffff:172.17.0.7:46162->:::53 10050 16:41:39.536837173 1 <NA> (36ae6d09d26e) skydns (17280:11) > recvmsg fd=6(<3t>:::53)

[…]

SkyDNS makes a request (10097) to /local/cluster/svc/critical-app/local/cluster/svc/critical-app/backend through the etcd API. Obviously etcd doesn’t recognize that service and returns (10167) a “Key not found”. This is passed back to curl via DNS query response.

[…]

10096 16:41:39.538247116 1 <NA> (36ae6d09d26e) skydns (4639:8) > write fd=3(<4t>10.0.2.15:34108->10.0.2.15:4001) size=221 10097 16:41:39.538275108 1 <NA> (36ae6d09d26e) skydns (4639:8) < write res=221 data=GET /v2/keys/skydns/local/cluster/svc/critical-app/local/cluster/svc/critical-app/backend?quorum=false&recursive=true&sorted=false HTTP/1.1Host: 10.0.2.15:4001User-Agent: Go 1.1 package httpAccept-Encoding: gzip10166 16:41:39.538636659 1 <NA> (36ae6d09d26e) skydns (4617:1) > read fd=3(<4t>10.0.2.15:34108->10.0.2.15:4001) size=4096 10167 16:41:39.538638040 1 <NA> (36ae6d09d26e) skydns (4617:1) < read res=285 data=HTTP/1.1 404 Not FoundContent-Type: application/jsonX-Etcd-Cluster-Id: 7e27652122e8b2aeX-Etcd-Index: 1259Date: Thu, 08 Dec 2016 15:41:39 GMTContent-Length: 112{“errorCode”:100,“message”:“Key not found”,“cause”:“/skydns/local/cluster/svc/critical-app/local”,“index”:1259}

[…]

curl doesn’t give up and tries again (10242) but this time with backend.critical-app.svc.cluster.local.svc.cluster.local. Looks like curl is trying a different search domain this time, as critical-app was removed from the appended domain. Of course, when forwarded to etcd (10274), this fails again (10345).

[…]

10218 16:41:39.538914765 0 client (b3a718d8b339) curl (22370:13) < connect res=0 tuple=172.17.0.7:35547->10.0.2.15:53 10242 16:41:39.539005618 1 <NA> (36ae6d09d26e) skydns (17280:11) < recvmsg res=74 size=74 data=backendcritical-appsvcclusterlocalsvcclusterlocal tuple=::ffff:172.17.0.7:35547->:::53 10247 16:41:39.539018226 1 <NA> (36ae6d09d26e) skydns (17280:11) > recvmsg fd=6(<3t>:::53) 10248 16:41:39.539019925 1 <NA> (36ae6d09d26e) skydns (17280:11) < recvmsg res=74 size=74 data=0]backendcritical-appsvcclusterlocalsvcclusterlocal tuple=::ffff:172.17.0.7:35547->:::53 10249 16:41:39.539022522 1 <NA> (36ae6d09d26e) skydns (17280:11) > recvmsg fd=6(<3t>:::53) 10273 16:41:39.539210393 1 <NA> (36ae6d09d26e) skydns (4639:8) > write fd=3(<4t>10.0.2.15:34108->10.0.2.15:4001) size=208 10274 16:41:39.539239613 1 <NA> (36ae6d09d26e) skydns (4639:8) < write res=208 data=GET /v2/keys/skydns/local/cluster/svc/local/cluster/svc/critical-app/backend?quorum=false&recursive=true&sorted=false HTTP/1.1

Host: 10.0.2.15:4001User-Agent: Go 1.1 package httpAccept-Encoding: gzip10343 16:41:39.539465153 1 <NA> (36ae6d09d26e) skydns (4617:1) > read fd=3(<4t>10.0.2.15:34108->10.0.2.15:4001) size=4096 10345 16:41:39.539467440 1 <NA> (36ae6d09d26e) skydns (4617:1) < read res=271 data=HTTP/1.1 404 Not Found[…]

curl will try once again, this time appending cluster.local as we can see the DNS query request (10418) to backend.critical-app.svc.cluster.local.cluster.local. This one (10479) obviously fails as well (10524), again.

[…]

10396 16:41:39.539686075 0 client (b3a718d8b339) curl (22370:13) < connect res=0 tuple=172.17.0.7:40788->10.0.2.15:53 10418 16:41:39.539755453 0 <NA> (36ae6d09d26e) skydns (17280:11) < recvmsg res=70 size=70 data=backendcritical-appsvcclusterlocalclusterlocal tuple=::ffff:172.17.0.7:40788->:::53 10433 16:41:39.539800679 0 <NA> (36ae6d09d26e) skydns (17280:11) > recvmsg fd=6(<3t>:::53) 10434 16:41:39.539802549 0 <NA> (36ae6d09d26e) skydns (17280:11) < recvmsg res=70 size=70 data=backendcritical-appsvcclusterlocalclusterlocal tuple=::ffff:172.17.0.7:40788->:::53 10437 16:41:39.539805177 0 <NA> (36ae6d09d26e) skydns (17280:11) > recvmsg fd=6(<3t>:::53) 10478 16:41:39.540166087 1 <NA> (36ae6d09d26e) skydns (4639:8) > write fd=3(<4t>10.0.2.15:34108->10.0.2.15:4001) size=204 10479 16:41:39.540183401 1 <NA> (36ae6d09d26e) skydns (4639:8) < write res=204 data=GET /v2/keys/skydns/local/cluster/local/cluster/svc/critical-app/backend?quorum=false&recursive=true&sorted=false HTTP/1.1Host: 10.0.2.15:4001User-Agent: Go 1.1 package httpAccept-Encoding: gzip10523 16:41:39.540421040 1 <NA> (36ae6d09d26e) skydns (4617:1) > read fd=3(<4t>10.0.2.15:34108->10.0.2.15:4001) size=4096 10524 16:41:39.540422241 1 <NA> (36ae6d09d26e) skydns (4617:1) < read res=267 data=HTTP/1.1 404 Not Found[…]

To the untrained eye, it might look that we have found the issue: a bunch of inefficient calls. But actually this is not true. If we look at the timestamps, the difference between the first etcd request (10097) and the last one (10479), the timestamps in the second column are less than 10ms apart. We are looking at an issue of seconds, not milliseconds - so where is the wait then?When we keep looking through the capture file, we can see that curl doesn’t stop trying with DNS queries to SkyDNS, now with backend.critical-app.svc.cluster.local.localdomain (10703). This .localdomain is not recognized by SkyDNS as an internal domain for Kubernetes so instead of going to etcd, it decides to forward this query to its upstream DNS resolver (10691).

[…]

10690 16:41:39.541376928 1 <NA> (36ae6d09d26e) skydns (4639:8) > connect fd=8(<4>) 10691 16:41:39.541381577 1 <NA> (36ae6d09d26e) skydns (4639:8) < connect res=0 tuple=10.0.2.15:44249->8.8.8.8:53 10702 16:41:39.541415384 1 <NA> (36ae6d09d26e) skydns (4639:8) > write fd=8(<4u>10.0.2.15:44249->8.8.8.8:53) size=68 10703 16:41:39.541531434 1 <NA> (36ae6d09d26e) skydns (4639:8) < write res=68 data=Nbackendcritical-appsvcclusterlocallocaldomain 10717 16:41:39.541629507 1 <NA> (36ae6d09d26e) skydns (4639:8) > read fd=8(<4u>10.0.2.15:44249->8.8.8.8:53) size=512 10718 16:41:39.541632726 1 <NA> (36ae6d09d26e) skydns (4639:8) < read res=-11(EAGAIN) data= 58215 16:41:43.541261462 1 <NA> (36ae6d09d26e) skydns (4640:9) > close fd=7(<4u>10.0.2.15:54272->8.8.8.8:53) 58216 16:41:43.541263355 1 <NA> (36ae6d09d26e) skydns (4640:9) < close res=0

[…]

Scanning down the timestamp column we see the first large gap when SkyDNS sends out a request and then hangs for about 4 seconds (10718-58215). Given that .localdomain is not a valid TLD (top level domain), the upstream server will be just ignoring this request. After the timeout, SkyDNS tries again with the same query (75923), hanging for another few more seconds (75927-104208). In total we have been waiting around 8 seconds for a DNS entry that doesn’t exist and is being ignored.

[…]

58292 16:41:43.542822050 1 <NA> (36ae6d09d26e) skydns (4640:9) < write res=68 data=Nbackendcritical-appsvcclusterlocallocaldomain 58293 16:41:43.542829001 1 <NA> (36ae6d09d26e) skydns (4640:9) > read fd=8(<4u>10.0.2.15:56371->8.8.8.8:53) size=512 58294 16:41:43.542831896 1 <NA> (36ae6d09d26e) skydns (4640:9) < read res=-11(EAGAIN) data= 75904 16:41:44.543459524 0 <NA> (36ae6d09d26e) skydns (17280:11) < recvmsg res=68 size=68 data=[…]

75923 16:41:44.543560717 0 <NA> (36ae6d09d26e) skydns (17280:11) < recvmsg res=68 size=68 data=Nbackendcritical-appsvcclusterlocallocaldomain tuple=::ffff:172.17.0.7:47441->:::53 75927 16:41:44.543569823 0 <NA> (36ae6d09d26e) skydns (17280:11) > recvmsg fd=6(<3t>:::53) 104208 16:41:47.551459027 1 <NA> (36ae6d09d26e) skydns (4640:9) > close fd=7(<4u>10.0.2.15:42126->8.8.8.8:53) 104209 16:41:47.551460674 1 <NA> (36ae6d09d26e) skydns (4640:9) < close res=0

[…]But finally, it all works! Why? curl stops trying to patch things and applying search domains. It tries the domain name verbatim as we typed in the command line. The DNS request is resolved by SkyDNS through the etcd API request (104406). A connection is opened against the service IP address (107992), then forwarded to the pod with iptables and the HTTP response travels back to the curl container (108024).

[…]

104406 16:41:47.552626262 0 <NA> (36ae6d09d26e) skydns (4639:8) < write res=190 data=GET /v2/keys/skydns/local/cluster/svc/critical-app/backend?quorum=false&recursive=true&sorted=false HTTP/1.1[…]

104457 16:41:47.552919333 1 <NA> (36ae6d09d26e) skydns (4617:1) < read res=543 data=HTTP/1.1 200 OK[…]

{“action”:“get”,“node”:{“key”:“/skydns/local/cluster/svc/critical-app/backend”,“dir”:true,“nodes”:[{“key”:“/skydns/local/cluster/svc/critical-app/backend/6ead029a”,“value”:“{\"host\”:\“10.3.0.214\”,\“priority\”:10,\“weight\”:10,\“ttl\”:30,\“targetstrip\”:0}“,"modifiedIndex”:270,“createdIndex”:270}],“modifiedIndex”:270,“createdIndex”:270}}[…]

107992 16:41:48.087346702 1 client (b3a718d8b339) curl (22369:12) < connect res=-115(EINPROGRESS) tuple=172.17.0.7:36404->10.3.0.214:80 108002 16:41:48.087377769 1 client (b3a718d8b339) curl (22369:12) > sendto fd=3(<4t>172.17.0.7:36404->10.3.0.214:80) size=102 tuple=NULL 108005 16:41:48.087401339 0 backend-1440326531-csj02 (730a6f492270) nginx (7203:6) < accept fd=3(<4t>172.17.0.7:36404->172.17.0.5:80) tuple=172.17.0.7:36404->172.17.0.5:80 queuepct=0 queuelen=0 queuemax=128 108006 16:41:48.087406626 1 client (b3a718d8b339) curl (22369:12) < sendto res=102 data=GET / HTTP/1.1[…]

108024 16:41:48.087541774 0 backend-1440326531-csj02 (730a6f492270) nginx (7203:6) < writev res=238 data=HTTP/1.1 200 OKServer: nginx/1.10.2[…]

Looking at how things operate at the system level we can conclude that there are two different issues as the root cause of this problem. First, curl doesn’t trust me when I give it a FQDN and tries to apply a search domain algorithm. Second, .localdomain should have never been there because it’s not routable within our Kubernetes cluster.If for a second you thought this could have been done using tcpdump, you haven’t tried yourself yet. I’m 100% sure is not going to be installed inside your container. You can run it outside from the host, but good luck finding the network interface matching the network namespace of the container that Kubernetes scheduled. If you don’t buy me, keep reading: we are not done with the troubleshooting yet.

Part2: DNS resolution troubleshooting

Let’s have a look at what’s in the resolv.conf file. The container could be gone already, or the file could have changed after the curl call.  But we have a sysdig capture that contains everything that happened.

Usually containers live as long as the process running inside them, disappearing when that process dies. This is one of the most challenging parts of troubleshooting containers. How we can explore something that’s gone already? How we can reproduce exactly what happened? Sysdig capture files come extremely useful in these cases.

Let’s analyze the capture file but instead of filtering the network traffic, we will filter on that file this time. We want to see resolv.conf exactly as it was read by curl, to confirm what we thought, it contains the localdomain.

$ sysdig -pk -NA -r capture.scap -c echo_fds “fd.type=file and fd.name=/etc/resolv.conf”—— Read 119B from  [k8s_client.eee910bc_client_critical-app_da587b4d-bd5a-11e6-8bdb-028ce2cfb533_bacd01b6] [b3a718d8b339]  /etc/resolv.conf (curl)

search critical-app.svc.cluster.local svc.cluster.local cluster.local localdomain

nameserver 10.0.2.15

options ndots:5

[…]

Here’s a new way to use sysdig:

-c echo_fds uses a Sysdig chisel - an add-on script - to aggregate the information and to format the output.

Also the filter includes only IO activity on file descriptors that are a file and with the name /etc/resolv.conf, exactly what we are looking for.Through the syscalls, we see there is an option called ndots. This option is the reason why curl didn’t trust our FQDN (fully qualified domain name) but tried to append all the search domain first. If you read the manpage, ndots forces libc that any resolution on a domain name with less than 5 dots won’t be treated as a fqdn but will try to first append all the search domains. ndots is there for a good reason, so we can perform a curl backend. But who added localdomain there?

Part3: Troubleshooting Docker containers run by Kubernetes

We don’t want to finish our troubleshooting without finding the culprit for this localdomain. That way, we can blame software and not people :) Was Docker who added that search domain? Or Kubernetes instructing Docker when creating the container?.

Since we know that all control communication between Kubernetes and Docker is done through a Unix socket, we can use that to filter things out:$ sudo sysdig -pk -s8192 -c echo_fds -NA “fd.type in (unix) and evt.buffer contains localdomain”

This time we will be capturing live with the help of an awesome filter, evt.buffer contains. This filter takes all the events buffers and if it contains the string we are looking for, will be considered for printing by our chisel that formats the output.

Now I need to create a new client to spy on what happens at container orchestration time:$ kubectl run -it –image=tutum/curl client-foobar –namespace critical-app –restart=NeverI can see that hyperkube, which is part of Kubernetes, wrote on /var/run/docker.sock using Docker API an HTTP POST request to /containers/create. If we read through it, we will find how this request contains an option “DnsSearch”:[“critical-app.svc.cluster.local”, “svc.cluster.local”, “cluster.local”, “localdomain”]. Kubernetes, we caught you!. Most probably it was there for some reason, like my local development machine having that search domain set up. In any case, that’s a different story.

[…]

—— Write 2.61KB to  [k8s-kubelet] [de7157ba23c4]   (hyperkube)POST /containers/create?name=k8s_POD.d8dbe16c_client-foobar_critical-app_085ac98f-bd64-11e6-8bdb-028ce2cfb533_9430448e HTTP/1.1Host: docker[…]

  "DnsSearch":[“critical-app.svc.cluster.local”,“svc.cluster.local”,“cluster.local”,“localdomain”],

[…]

Conclusion

Reproducing exactly what happened inside container can be very challenging as they terminate when the process dies or just ends. Sysdig captures contain all the information through the system calls including network traffic, file system I/O and processes behaviour providing all the data required for troubleshooting.

When troubleshooting in a container environment, being able to filter and add container contextual information like Docker container names or Kubernetes metadata makes our lives significantly easier.

Sysdig is available in all the main Linux distros, for OSX and also Windows. Download it from here to get the last version. Sysdig is an open source tool but the company behind the project also offers a commercial product to monitor and troubleshoot containers and microservices across multiple hosts.

Day 6 - Apache Mesos and the Rise of the Datacenter OS

Written by: Roger Ignazio (@rogerignazio)
Edited by: Justin Garrison (@rothgar)

Roger Ignazio is an Infrastructure Automation Engineer at Mesosphere and the author of "Mesos In Action." Thanks to the generosity of the team at Manning Publications, SysAdvent readers enjoy a 40% discount when they use the code “mesysad” at https://manning.com/books/mesos-in-action.

Containers and application orchestration are hot topics as organizations and engineering teams attempt to deploy changes to applications and infrastructure as quickly as possible, all while improving overall datacenter efficiency. When you read articles about containers, mentions of Apache Mesos (paper) usually aren’t too far away. You may be wondering what Mesos is, and how to use it for managing applications at scale.

In this article, I’ll provide an introduction to Mesos, drawing a number of comparisons between the Linux kernel and Mesos to help you understand how it works. I’ll cover two open source projects that allow engineering teams to quickly and easily deploy applications and scheduled tasks on a cluster. And finally, I’ll discuss how Mesosphere is combining all of this to create a datacenter-wide operating system, with Mesos at its core.

Mesos – A Distributed Kernel

Whether you’re reading this post on your laptop, smartphone, or tablet, chances are that you have no idea which core of the processor your web browser is using. Sure, you could find out, but why bother? The operating system’s kernel handles the resource abstraction and scheduling for you. In the end, the operating system probably doesn’t even matter all that much; you just want a way to run the apps you love, and the operating system is a means to provide that experience.

Mesos, not unlike the kernel of an operating system, provides a way to abstract resources from physical or virtual machines. But where it begins to differ from OS kernels (such as Linux) is that the abstraction isn’t bound to a single host. Instead, Mesos provides a way to abstract resources for any number of machines—from 10s to over 10,000—and program them as a single entity, leading to simplified systems management and improved resource utilization. Many companies such as Apple, Twitter, Airbnb, Bloomberg, and others have turned to Mesos to power their computing infrastructure.

To better visualize what I just explained, take a look at the following graphic comparing the resource abstraction and scheduling between a single machine running the Linux kernel, and multiple machines taking part in a Mesos cluster.

In both cases, some abstraction layer—whether it’s single machine hardware with the Linux kernel or multiple machines with Mesos—is responsible for offering compute resources (processor cores, memory, storage, and network ports) to applications. Linux does a fantastic job of doing this on a single machine, but what happens when you want to deploy applications to multiple machines? Each machine effectively becomes a silo, only able to provide resources to applications that also run on that box.

Resource scheduling

Despite being a popular buzzword lately, "resource scheduling" is anything but a new concept. Two popular examples are the Completely Fair Scheduler in the Linux kernel and the Distributed Resource Scheduler (DRS) in VMware vSphere. In both cases, these schedulers seek to optimize the scheduling of tasks based on the available resources. Mesos borrows ideas from both of these schedulers and builds them into an abstraction layer, or "distributed kernel" of its own. But unlike the Linux kernel, which primarily provides access to underlying physical (or virtual) compute resources on a single machine, Mesos agents offer resources to a Mesos master, to then be consumed by various applications.

The Mesos master implements a two-tier scheduling model, which allows the Mesos master to send resource offers to an application (or framework, in Mesos terms). The application can then accept or decline the offer, based on the attributes in the offer, or if it has any tasks to be launched. The Mesos master is able to schedule multiple different resource types—CPUs, memory, disk, and ports—among various different applications by using the Dominant Resource Fairness algorithm built into its resource allocation module. So instead of provisioning a number of machines to run a specific service, you’re now able to define the amount of resources an application needs, and allow it to be scheduled anywhere on the cluster.

The end of static partitioning

Let’s take a step back for a moment and consider the following scenario. You have two services in your datacenter: a Ruby on Rails application, and a Jenkins CI cluster. When one of these services requires more resources than a single machine can provide, you provision more machines. If your Rails app needs to handle additional users, you’re required to provision a new server and [re]configure your load balancer. If builds are being queued up by your Jenkins CI master, you provision an additional Jenkins agent and manually attach it to the master.

That scenario suggests that humans are being left to perform resource allocation and capacity planning by hand, and frankly, we’re pretty bad at it. Chances are that each of those applications doesn’t operate at 100% utilization 100% of the time, leading to a disappointing industry-average 6-12% system utilization. When you start measuring overall utilization against operating costs, that’s a lot of wasted capital! Regardless of whether you’re running your own datacenter or using an Infrastructure-as-a-Service provider like AWS or Azure, they’re all servers with unused cycles at the end of the day.

By abstracting system resources, you’re able to stop guessing at the number of machines required for a specific application and instead focus on the amount of resources they’re actually consuming—that is, the amount of CPUs and memory that it requires to run your Rails app or your CI service. As long as the number of machines in the datacenter provide enough resources to run all of your workloads, you’re in good shape. And if not, the workloads can be queued until resources become available. But the key takeaway is that you’re now adding additional compute resources to the larger Mesos cluster and not to a number of small, statically-partitioned services or manually creating and configuring VMs.

Containerization

In a way that draws many parallels to the rise of the intermodal shipping container and the containerization movement of the freight industry during the 20th century, we’re seeing the same movement in the IT industry, but at a staggering pace. In fact, the use of containers in computing isn’t exactly new either; control groups (or cgroups, for short) was first added to the Linux kernel back in January 2008 as a way to isolate individual processes. In the last couple of years, Docker has made it incredibly simple for end-users to get started with this technology.

Because containers provide a lightweight alternative to virtual machines, allowing users to run their applications in isolated environments, Mesos is built with containers at its core, supporting both Linux cgroups and Docker containers. Each of these container technologies allow for tasks to run at varying levels of isolation from other tasks on the same machine.

But being a distributed kernel, Mesos only provides a means to launch processes using these container technologies and handles things like resource allocation and port and volume mappings. To launch tasks using Mesos we need an init system to manage the tasks and it would be nice to have a cron system to go with it.

Marathon and Chronos – Init and Cron Frameworks

I already mentioned that Mesos provides a way for multiple applications (or frameworks to use Mesos terms) to share multiple different types of resources on a given cluster. A framework registers with the Mesos master and receives resource offers. There are a number of different frameworks currently available, including big data processing (Spark, Kafka), distributed storage and databases (HDFS, Cassandra), batch scheduling (Chronos, Aurora), and long-running services (Marathon, Aurora). I’ll focus on two popular frameworks: Marathon and Chronos. These frameworks are used to deploy long-running services (like web apps) and distributed cron jobs, respectively.

Marathon

Marathon is an open source init system for Mesos developed by Mesosphere. It’s roughly equivalent to supervisord in that it manages long-running tasks and automatically restarts application instances if one of them should fail. So if a machine in your cluster fails in the middle of the night, Marathon automatically reschedules the failed applications on an available machine.

Marathon supports launching both cgroups and Docker containers on a Mesos cluster, and can quickly and easily scale an application up to N instances.

Marathon also includes an extensive REST API which allows you to create, modify, and delete applications, and query the service for information about running instances. This allows you to automatically perform rolling upgrades of your application using your CI system, or to dynamically create HAProxy configurations and reload the service when changes have occurred. When it comes to application management, Marathon allows you to take the worst part of your scheduling, the human, out of your infrastructure.

Chronos

Chronos is an open source cron system for Mesos originally developed at Airbnb. It builds upon traditional cron with features such as ISO 8601-formatted timestamps, automatic retries of failed jobs, specifying a maximum number of times a task should run, and the ability for a job to have dependencies on other jobs. Like Marathon, it also supports running tasks in cgroups and Docker containers and provides an extensive REST API that can be used for creating, modifying, deleting, and manually triggering jobs.

Mesosphere DCOS – A Mesos-based Operating System

If you take a minute to think about the components that make up an operating system such as such as Red Hat Enterprise Linux or Ubuntu you’d probably identify the following:

• Init system: A daemon (PID 1) such as Systemd (RHEL 7) or Upstart (Ubuntu 14.04) manage long-running services and can automatically restart (or respawn) services if and when they fail.

• Package management: A package format (rpm, deb), package manager (yum, apt), and a set of base repositories (base, main).

• Command line interface: A shell that is launched when a user logs in (bash, zsh).

• Graphical user interface: An optional graphical user interface for monitoring and administering the system.

With the open source Mesos project as its distributed kernel and Marathon as its init system, Mesosphere has set out to build a modern, distributed, enterprise-grade operating system. This system, appropriately named the Datacenter Operating System (DCOS), provides a way for systems administrators to deploy applications and services at scale without needing to worry about things like statically partitioning services or machines failing in the middle of the night. DCOS is currently offered in two flavors: Enterprise and Community.

Package management

At the time of this writing, Mesosphere provides two package repositories for the DCOS: Universe and Multiverse. These repositories host production-ready and beta packages respectively. The documentation for the Universe covers the schema quite nicely so I won’t cover it all here, but it essentially boils down to a package definition being a JSON object that can be processed by the DCOS CLI and understood by Marathon’s API.

Command line interface

The DCOS CLI can be installed on your laptop or workstation and interacts with various services in DCOS. It provides functionality for managing packages, services, and nodes in a DCOS cluster.

SERVICES=( chronos jenkins spark hdfs cassandra kubernetes )
for service in ${SERVICES[@]}; do
    dcos package install --yes $service
done

Some of these services—Cassandra, HDFS, Kubernetes—require non-trivial amounts of effort to deploy effectively. The team at Mesosphere, using these package repositories, provides and maintains turn-key solutions for deploying these services in your own datacenter in a fully automated, fault-tolerant manner.

Graphical user interface

Although the DCOS CLI allows you to fully administer the operating system from the command line, the web interface provides information about the cluster including installed services, running tasks, and nodes belonging to the cluster.

To deploy your own applications and Docker containers you can use the CLI or navigate to the Marathon web interface (available on the Services tab). There you can create a new application specifying the required resources the number of instances the information for the Docker container image, etc. Although the DCOS includes Marathon as its init system, it’s also possible to deploy multiple instances of Marathon on top of Marathon so that you can provide individual teams with their own Platform-as-a-Service without worrying about them affecting another team’s applications.

Summary

Mesos provides a layer of abstraction for the resources on many machines in a datacenter allowing them to be programmed as a single entity. It allows multiple applications to share a single cluster of machines without worrying about statically partitioning services within the datacenter. Marathon allows you to deploy applications and long-running services on the cluster, and the information available via its API can be used to dynamically create load balancer configurations and reload the configuration when changes occur. Mesosphere DCOS combines a number of open source and commercial components into an easy to manage and deploy Mesos cluster, allowing you to quickly deploy applications and containers.

So, if you’re looking to improve the resource utilization of your own infrastructure and get rid of your human scheduling bottleneck, or if you’re just looking for something new to play with, maybe it’s time to give Mesos and Mesosphere DCOS a try.

• https://github.com/mesosphere/playa-mesos

• https://docs.mesosphere.com/getting-started