Day 6 - Apache Mesos and the Rise of the Datacenter OS

Written by: Roger Ignazio (@rogerignazio)
Edited by: Justin Garrison (@rothgar)

Roger Ignazio is an Infrastructure Automation Engineer at Mesosphere and the author of "Mesos In Action." Thanks to the generosity of the team at Manning Publications, SysAdvent readers enjoy a 40% discount when they use the code “mesysad” at https://manning.com/books/mesos-in-action.

Containers and application orchestration are hot topics as organizations and engineering teams attempt to deploy changes to applications and infrastructure as quickly as possible, all while improving overall datacenter efficiency. When you read articles about containers, mentions of Apache Mesos (paper) usually aren’t too far away. You may be wondering what Mesos is, and how to use it for managing applications at scale.

In this article, I’ll provide an introduction to Mesos, drawing a number of comparisons between the Linux kernel and Mesos to help you understand how it works. I’ll cover two open source projects that allow engineering teams to quickly and easily deploy applications and scheduled tasks on a cluster. And finally, I’ll discuss how Mesosphere is combining all of this to create a datacenter-wide operating system, with Mesos at its core.

Mesos – A Distributed Kernel

Whether you’re reading this post on your laptop, smartphone, or tablet, chances are that you have no idea which core of the processor your web browser is using. Sure, you could find out, but why bother? The operating system’s kernel handles the resource abstraction and scheduling for you. In the end, the operating system probably doesn’t even matter all that much; you just want a way to run the apps you love, and the operating system is a means to provide that experience.

Mesos, not unlike the kernel of an operating system, provides a way to abstract resources from physical or virtual machines. But where it begins to differ from OS kernels (such as Linux) is that the abstraction isn’t bound to a single host. Instead, Mesos provides a way to abstract resources for any number of machines—from 10s to over 10,000—and program them as a single entity, leading to simplified systems management and improved resource utilization. Many companies such as Apple, Twitter, Airbnb, Bloomberg, and others have turned to Mesos to power their computing infrastructure.

To better visualize what I just explained, take a look at the following graphic comparing the resource abstraction and scheduling between a single machine running the Linux kernel, and multiple machines taking part in a Mesos cluster.

In both cases, some abstraction layer—whether it’s single machine hardware with the Linux kernel or multiple machines with Mesos—is responsible for offering compute resources (processor cores, memory, storage, and network ports) to applications. Linux does a fantastic job of doing this on a single machine, but what happens when you want to deploy applications to multiple machines? Each machine effectively becomes a silo, only able to provide resources to applications that also run on that box.

Resource scheduling

Despite being a popular buzzword lately, "resource scheduling" is anything but a new concept. Two popular examples are the Completely Fair Scheduler in the Linux kernel and the Distributed Resource Scheduler (DRS) in VMware vSphere. In both cases, these schedulers seek to optimize the scheduling of tasks based on the available resources. Mesos borrows ideas from both of these schedulers and builds them into an abstraction layer, or "distributed kernel" of its own. But unlike the Linux kernel, which primarily provides access to underlying physical (or virtual) compute resources on a single machine, Mesos agents offer resources to a Mesos master, to then be consumed by various applications.

The Mesos master implements a two-tier scheduling model, which allows the Mesos master to send resource offers to an application (or framework, in Mesos terms). The application can then accept or decline the offer, based on the attributes in the offer, or if it has any tasks to be launched. The Mesos master is able to schedule multiple different resource types—CPUs, memory, disk, and ports—among various different applications by using the Dominant Resource Fairness algorithm built into its resource allocation module. So instead of provisioning a number of machines to run a specific service, you’re now able to define the amount of resources an application needs, and allow it to be scheduled anywhere on the cluster.

The end of static partitioning

Let’s take a step back for a moment and consider the following scenario. You have two services in your datacenter: a Ruby on Rails application, and a Jenkins CI cluster. When one of these services requires more resources than a single machine can provide, you provision more machines. If your Rails app needs to handle additional users, you’re required to provision a new server and [re]configure your load balancer. If builds are being queued up by your Jenkins CI master, you provision an additional Jenkins agent and manually attach it to the master.

That scenario suggests that humans are being left to perform resource allocation and capacity planning by hand, and frankly, we’re pretty bad at it. Chances are that each of those applications doesn’t operate at 100% utilization 100% of the time, leading to a disappointing industry-average 6-12% system utilization. When you start measuring overall utilization against operating costs, that’s a lot of wasted capital! Regardless of whether you’re running your own datacenter or using an Infrastructure-as-a-Service provider like AWS or Azure, they’re all servers with unused cycles at the end of the day.

By abstracting system resources, you’re able to stop guessing at the number of machines required for a specific application and instead focus on the amount of resources they’re actually consuming—that is, the amount of CPUs and memory that it requires to run your Rails app or your CI service. As long as the number of machines in the datacenter provide enough resources to run all of your workloads, you’re in good shape. And if not, the workloads can be queued until resources become available. But the key takeaway is that you’re now adding additional compute resources to the larger Mesos cluster and not to a number of small, statically-partitioned services or manually creating and configuring VMs.

Containerization

In a way that draws many parallels to the rise of the intermodal shipping container and the containerization movement of the freight industry during the 20th century, we’re seeing the same movement in the IT industry, but at a staggering pace. In fact, the use of containers in computing isn’t exactly new either; control groups (or cgroups, for short) was first added to the Linux kernel back in January 2008 as a way to isolate individual processes. In the last couple of years, Docker has made it incredibly simple for end-users to get started with this technology.

Because containers provide a lightweight alternative to virtual machines, allowing users to run their applications in isolated environments, Mesos is built with containers at its core, supporting both Linux cgroups and Docker containers. Each of these container technologies allow for tasks to run at varying levels of isolation from other tasks on the same machine.

But being a distributed kernel, Mesos only provides a means to launch processes using these container technologies and handles things like resource allocation and port and volume mappings. To launch tasks using Mesos we need an init system to manage the tasks and it would be nice to have a cron system to go with it.

Marathon and Chronos – Init and Cron Frameworks

I already mentioned that Mesos provides a way for multiple applications (or frameworks to use Mesos terms) to share multiple different types of resources on a given cluster. A framework registers with the Mesos master and receives resource offers. There are a number of different frameworks currently available, including big data processing (Spark, Kafka), distributed storage and databases (HDFS, Cassandra), batch scheduling (Chronos, Aurora), and long-running services (Marathon, Aurora). I’ll focus on two popular frameworks: Marathon and Chronos. These frameworks are used to deploy long-running services (like web apps) and distributed cron jobs, respectively.

Marathon

Marathon is an open source init system for Mesos developed by Mesosphere. It’s roughly equivalent to supervisord in that it manages long-running tasks and automatically restarts application instances if one of them should fail. So if a machine in your cluster fails in the middle of the night, Marathon automatically reschedules the failed applications on an available machine.

Marathon supports launching both cgroups and Docker containers on a Mesos cluster, and can quickly and easily scale an application up to N instances.

Marathon also includes an extensive REST API which allows you to create, modify, and delete applications, and query the service for information about running instances. This allows you to automatically perform rolling upgrades of your application using your CI system, or to dynamically create HAProxy configurations and reload the service when changes have occurred. When it comes to application management, Marathon allows you to take the worst part of your scheduling, the human, out of your infrastructure.

Chronos

Chronos is an open source cron system for Mesos originally developed at Airbnb. It builds upon traditional cron with features such as ISO 8601-formatted timestamps, automatic retries of failed jobs, specifying a maximum number of times a task should run, and the ability for a job to have dependencies on other jobs. Like Marathon, it also supports running tasks in cgroups and Docker containers and provides an extensive REST API that can be used for creating, modifying, deleting, and manually triggering jobs.

Mesosphere DCOS – A Mesos-based Operating System

If you take a minute to think about the components that make up an operating system such as such as Red Hat Enterprise Linux or Ubuntu you’d probably identify the following:

• Init system: A daemon (PID 1) such as Systemd (RHEL 7) or Upstart (Ubuntu 14.04) manage long-running services and can automatically restart (or respawn) services if and when they fail.

• Package management: A package format (rpm, deb), package manager (yum, apt), and a set of base repositories (base, main).

• Command line interface: A shell that is launched when a user logs in (bash, zsh).

• Graphical user interface: An optional graphical user interface for monitoring and administering the system.

With the open source Mesos project as its distributed kernel and Marathon as its init system, Mesosphere has set out to build a modern, distributed, enterprise-grade operating system. This system, appropriately named the Datacenter Operating System (DCOS), provides a way for systems administrators to deploy applications and services at scale without needing to worry about things like statically partitioning services or machines failing in the middle of the night. DCOS is currently offered in two flavors: Enterprise and Community.

Package management

At the time of this writing, Mesosphere provides two package repositories for the DCOS: Universe and Multiverse. These repositories host production-ready and beta packages respectively. The documentation for the Universe covers the schema quite nicely so I won’t cover it all here, but it essentially boils down to a package definition being a JSON object that can be processed by the DCOS CLI and understood by Marathon’s API.

Command line interface

The DCOS CLI can be installed on your laptop or workstation and interacts with various services in DCOS. It provides functionality for managing packages, services, and nodes in a DCOS cluster.

SERVICES=( chronos jenkins spark hdfs cassandra kubernetes )
for service in ${SERVICES[@]}; do
    dcos package install --yes $service
done

Some of these services—Cassandra, HDFS, Kubernetes—require non-trivial amounts of effort to deploy effectively. The team at Mesosphere, using these package repositories, provides and maintains turn-key solutions for deploying these services in your own datacenter in a fully automated, fault-tolerant manner.

Graphical user interface

Although the DCOS CLI allows you to fully administer the operating system from the command line, the web interface provides information about the cluster including installed services, running tasks, and nodes belonging to the cluster.

To deploy your own applications and Docker containers you can use the CLI or navigate to the Marathon web interface (available on the Services tab). There you can create a new application specifying the required resources the number of instances the information for the Docker container image, etc. Although the DCOS includes Marathon as its init system, it’s also possible to deploy multiple instances of Marathon on top of Marathon so that you can provide individual teams with their own Platform-as-a-Service without worrying about them affecting another team’s applications.

Summary

Mesos provides a layer of abstraction for the resources on many machines in a datacenter allowing them to be programmed as a single entity. It allows multiple applications to share a single cluster of machines without worrying about statically partitioning services within the datacenter. Marathon allows you to deploy applications and long-running services on the cluster, and the information available via its API can be used to dynamically create load balancer configurations and reload the configuration when changes occur. Mesosphere DCOS combines a number of open source and commercial components into an easy to manage and deploy Mesos cluster, allowing you to quickly deploy applications and containers.

So, if you’re looking to improve the resource utilization of your own infrastructure and get rid of your human scheduling bottleneck, or if you’re just looking for something new to play with, maybe it’s time to give Mesos and Mesosphere DCOS a try.

• https://github.com/mesosphere/playa-mesos

• https://docs.mesosphere.com/getting-started

Day 5 - ELK Operations and Administration

Written by: Ryn Daniels (@rynchantress)
Edited by: Ben Cotton (@funnelfiasco)

Introduction

If you’ve ever had the misfortune of using a computer before, you’ve probably encountered what it’s like when a computer does something wrong. When you have a lot of computers working together doing a lot of computer-things (several to a lot of those things being wrong), it helps to have logs of what’s been going on to help with troubleshooting, and it really helps to have those logs in some kind of centralized location. And if you’ve ever run into this kind of situation at work, then you’ve probably had to figure out how to set up and manage some kind of logging solution.

When I was in just this situation in a previous lifetime, back when I was a one-person ops team, I was struggling with the build versus buy versus open-source decision. I really wanted to try to set up an ELK (Elasticsearch, Logstash, and Kibana) cluster, but given that I was a one-person team, I didn’t have the time or the resources to figure it all out on my own from scratch. The information that I found online seemed mostly aimed at developers, not ops folks, none of the community Chef cookbooks worked off the shelf, and with everything else on my plate I didn’t have the time to make any of it work for me.

But now I’m at Etsy working on an incredible ops team with enough resources to dedicate to ELK wrangling to get it right, and we’ve gotten it operationalized and automated to the point that it’s very nearly as easy as “press button get ELK cluster.” My goal for this post is to write the one I wish I could have read so many years ago, with the hope that it will be helpful to anyone else who has to set up and manage an ELK cluster. This won’t cover what ELK is, how it compares to other logging solutions, or how to get it set up manually - those topics have all been covered elsewhere. What I will talk about is how we have our clusters set up in Chef, how we monitor and manage the clusters on a day-to-day basis, and how we handle things when the herd of ELK goes all sideways.

Press Button, Get ELK Cluster: Choosing and Provisioning Hardware

At Etsy, we use ELK pretty extensively for the vast majority of our log storage and searching needs. Because snowflake servers make us sad, all of our ELK roles are in Chef. In this repo you can see examples of the roles, recipes, templates, and data bags that we use to create a cluster. With this combination of roles and recipes, building a new cluster is as simple as provisioning nodes with the Elasticsearch, Logstash, and Kibana roles - when they come up, all the pieces are already working together and ready for log ingestion. For our large production cluster, each of the E, L, and K components is on its own dedicated hardware, but if your hardware is limited, all three roles can be combined on the same servers.

A few things that we’ve learned over the years with regards to provisioning ELK clusters:

• Hardware matters! Specifically, disk IO is very important to ELK performance. We’re using SSDs for all of our ELK clusters. If you have your own data centers and are also using SSDs, make sure you choose SSDs with enough write performance to handle the load you expect on your cluster (this is often noted as something like DWPD or disk writes per day) - SSDs that are designed to handle read-intensive loads will die a lot earlier if you are writing hundreds of GB per day to them. As an example, we ingest between 7 and 10 TB of logs daily, and keep them available for 30 days - to support this without falling over, our production cluster has 70 Elasticsearch, 10 Logstash, and 2 Kibana servers, with the Elasticsearch servers using SSDs that are rated for write-intensive workloads.
• Network bandwidth matters too! With a main production cluster that ingests a several TB of logs per day, we found that we absolutely had to have 10GbE network cards in these servers rather than 1GbE cards. This was for making sure that node failures didn’t cause data loss - with our log volume, recovery from a node failure with a 1gig NIC would take multiple to tens of hours to recover, slowing down the cluster’s performance as well as increasing the window of time during which a second node failure might cause data loss. With 10gig NICs, recovery and rebalancing operations take much less time - again this is with our 7-10TB of logs per day, so if your expected daily and overall log volume are much lower, 1gig might be fine for you. This applies in the cloud too - if you’re using something like AWS, make sure you pick instances with enough network bandwidth available to handle your expected load.
• By default, the Elasticsearch template that ships with Logstash will pull out and index every single possible field in every log line - we found that this had a noticeable negative impact on cluster performance with our log volume, and since we generally had only a few fields that we really needed to index on, decided to Chef out a new template that turned off this default behavior. Your mileage may vary, but this is something else to look out for if you’re expecting to have a relatively high log volume (anything under a terabyte daily and you’ll probably be fine with most of the defaults).

We also use Chef to install the Lumberjack log forwarder (our internal fork of logstash-forwarder) to client nodes that need to get logs into logstash. The recipe to install and set up Lumberjack is also found in the ELK-utils repo, and then log files are configured to be ingested by declaring them as attributes in roles:

"lumberjack_files": [
  {
      "paths": [ "/var/log/httpd/access_log" ],
      "fields": { "type": "web_access_log" }
  }
]

Adding a new log and index to one of our existing ELK clusters involves making changes in three places: The Chef role, where the log and its type are specified, the logstash.json configuration file, which is cheffed out in our Logstash cookbook and defines which grok patterns and indices are used for that log type, and the grok pattern (if a new one is being defined), which is also part of the Logstash cookbook. If you’re ever writing grok patterns, the Grok Debugger is incredibly helpful - don’t go elk herding without it!

With everything in Chef, getting up and running with a new cluster is something that can happen in a matter of hours, from building the boxes to searching for your newly ingested logs in Kibana. But that’s not all that has to be considered - ELK clusters generally need a fair amount of love to keep them up and running happily.

Day to Day Operations: Chatops to the Rescue

Because we’re big into IRC and chatops at Etsy, we’ve taught our friendly local irccat a few things about ELK so that we can use it to help troubleshoot and manage the cluster in chat. Most of the stability and maintenance issues we’ve run into so far have involved the Elasticsearch part of the stack, so most of our tooling has been built to help us with that aspect. One of the nice things about Elasticsearch is its API, but one of the less-than-nice things about that API is having to remember all the different endpoints you might need to curl to get the information you need. Luckily, computers are here to remember things for us!

We approached this by taking the most common API endpoints we found ourselves using and writing irccat commands to do them for us. You might find that you end up using different ones more often, but some of the commands that we’ve created and found very helpful include:

Seeing the current health of the cluster

Finding which of the eligible servers is the current master

Getting a list of the current indices in general…

... or getting more detailed information about indices with a given name, including color-coded output for easier readability

Listing any current recoveries ongoing in the cluster

Seeing the status of all the shards in the cluster

We also have the ability to modify a few settings from IRC, such as the maximum number of concurrent rebalancing operations that can be in progress at any one time - we’ll normally have this set relatively low, to 1 or 2, but when the cluster is recovering from some event or we’re adding new nodes to it, we’ll turn this up to 7 or 8 to allow the cluster the rebalance itself as quickly as possible.

When Things Go Sideways: Monitoring, Troubleshooting, and Maintenance

With this combination of commands, as well as dashboards that pull data from both Ganglia and Graphite to give us near-realtime looks into how each of the various sub-clusters of our ELK cluster are doing, we’ve found ourselves able to quite handily troubleshoot most failure cases that have come our way so far. ?elk master quickly lets us know which server is the current Elasticsearch master, giving us a good starting point for troubleshooting - most often in the elasticsearch logs for the master ES process. Those logs will point us to individual servers in the cluster that are causing problems - the bulk of the issues we’ve seen so far have been the cluster taking too long to mark unresponsive nodes as failed and remove them from the cluster, which was fixed with our upgrade to Elasticsearch 1.5.2. With this upgrade we also saw the cluster healing faster in general, so if you’re starting a new cluster, we’d highly recommend going with the latest and greatest stable version from the get-go.

Our dashboards are created using Etsy’s own dashboard framework and are used to view both system-level metrics, collected via Ganglia, and application-level metrics that are gathered and put into Graphite.

While we did experiment with using Elasticsearch’s Marvel plugin for monitoring, we ended up not being too keen on having Elasticsearch running in order to monitor Elasticsearch - having our monitoring be completely decoupled from what is being monitored has been working much better for us, and we’re quite happily using things like Elasticsearch’s API to generate all the metrics we need, but Marvel is certainly worth a look - your mileage may vary.

The one major issue we’ve had relates to SSDs - a firmware upgrade was required on every single SSD in our production Elasticsearch cluster - and we wanted to get that done without any downtime! The manufacturer we were working with could only replace 66 disks at a time, so with 6 disks per ES node, we were cycling batches of 11 servers in and out of the cluster - just under 8 batches in all. This has been the biggest operational headache we’ve had in terms of managing ELK in production, but the process of working through that issue led to the creation of the ELK-utils repository and many of our ELK management irccat commands - so it wasn’t a complete disaster!

How big of a headache was this? Elasticsearch lets you specify which nodes in a cluster are included and excluded for a given index with a simple API call, so to remove a server from the cluster, all we had to do was add that server to the exclude list for… every single index. With close to 50 indices in the cluster at a time (we index our production logs daily) and upwards of 70 servers, math shows us that that’s a lot of API calls to make. So we turned to our faithful friend automation to do it for us, and thus the elkvacuate script was born. With this script we were able to generate and execute all those API calls automatically, so 500+ curl commands was reduced to 11 runs of this script to add or remove one batch of servers (we purposely removed servers one at a time to avoid overwhelming the cluster with rebalancing operations, which happened when we tried doing 11 at once).

Conclusion

There’s a lot I covered in here, and even more that I didn’t cover, because it would take a blog post the size of an elk to go into detail on everything related to managing and operating an ELK cluster. If I could leave you with a few pieces of wisdom, they would be: pay attention to your (expected) capacity before you start setting things up, monitoring and configuration management are your friends, and JSON-based APIs are a lot friendlier when you can automate away most of the nitty gritty details.

Thank You

Special thanks to Ben Cotton for editing, Ryan Frantz for additional editing and general awesomeness, and everyone who read this! If you’d like to chat more about shaving ELK-yaks, you can find me on the internets.

Day 4 - End to End Deployment with Microsoft Azure

Written by: @samcogan
Edited by: @phrawzty

If you're a sysadmin who needs to deploy resources in Microsoft's Azure platform your choices have been limited to either the web portal or complex scripts. Today I want to highlight a set of new technologies that when combined together provide a way to move away from manual processes and complex scripts and instead produce a declarative, reusable, and easy to manage and maintain solution for deployment to Azure. Using these tools we will be able to stop focussing on defining "how" we deploy, and instead start defining recipes for “what” we want to deploy and let the technology take care of “how”. By the time you’ve finished this article you’ll see the power of these technologies and the agility they can bring to your deployment process.

Before we dive in, let’s take a step back and understand what the problem is we are trying to solve. Deploying resources to any cloud provider usually involves more than a few moving parts such as storage, virtual machines, websites, and databases -all with dependencies on each other. This gets even more complex when talking about test or development environments where there is a need to regularly create and destroy environments.

In this sort of world what is needed is a way to easily define an environment, automate the deployment and tear down of that environment, and then do it all again with the same results. Additionally you also want to apply software development practices such as testing, version control, patterns and practices to your deployment process - collectively, this is often referred to as "Infrastructure as Code" (IaC).

Up until recently this has been difficult (if not impossible) to do with Microsoft’s Azure platform. Sure you could create PowerShell scripts to deploy resources, but this all happened in a serial fashion with each resource having no knowledge or dependencies on other resources. This was more like "Infrastructure as scripts that failed as often as they worked". This lagged behind platforms such as AWS where solutions like Cloudformation offer a relatively straightforward way to represent your infrastructure as JSON.

This has all changed in the last year with the release of new tools and enhancement of existing ones to allow an Infrastructure as code approach to Azure deployments.

Azure Resource Manager

Azure Resource Manager (ARM) is the first new feature that enables moving to a code based deployment. ARM introduces a number of features, but the ones key to an IaC approach are:

• Creation of resources inside a resource group so they can be managed collectively.

• Declarative templates to define deployments.

• Idempotent deployments that can be re-applied to ensure a consistent state.

• Deployment of non-dependant resources in parallel to improve deployment times.

With these four components it is now possible to create recipes for your deployments using JSON templates. These templates define both the resources themselves and the relationships between them.

Once your templates are finished you can then deploy these to a resource group. If this is the first time you have deployed to this resource group it will create your resources; however, if you're running against an existing deployment it will validate the state of the deployment and amend or re-deploy resources as required.

Finally, if you want to delete your deployment, all you need to do is delete the resource group and that will delete and cleanup all the resources contained within. If you want to re-deploy, just run the deployment again.

Because your deployment is now in a declarative template you can store this in your version control tool, add it to your testing process, and use them as part of your build process to create test environments. You can also use Visual Studio to create your templates (ensure you have the latest Azure SDK and PowerShell tools for Visual Studio installed).

Templates are not limited solely to deploying top level resources! For example, a template that deploys an Azure Website can include a resource that downloads an MS Deploy package and deploys it to the website. Similarly, with a virtual machine you can use ARM to install various extension to provide things like anti virus protection, monitoring, or further configuration of the VM which we will discuss in the next section. Follow the link for the Azure Quick Start Templates page in the resources section to see examples of what can be achieved with Azure Resource Manager.

Below is an example of a ARM resource that deploys a website and downloads and applies an MS Deploy package for the content.

"resources": [
    {
      "apiVersion": "2014-06-01",
      "type": "Microsoft.Web/serverfarms",
      "name": "[parameters('hostingPlanName')]",
      "location": "[resourceGroup().location]",
      "properties": {
          "name": "[parameters('hostingPlanName')]",
          "sku": "[parameters('hostingPlanSku')]",
          "workerSize": "0",
          "numberOfWorkers": 1
      }
    },
    {
      "apiVersion": "2014-06-01",
      "type": "Microsoft.Web/sites",
      "name": "[parameters('siteName')]",
      "location": "[resourceGroup().location]",
      "tags": {
          "environment": "test",
      },
      "dependsOn": [
          "[resourceId('Microsoft.Web/serverfarms', parameters('hostingPlanName'))]"
      ],
      "properties": {
          "name": "[parameters('siteName')]",
          "serverFarm": "[parameters('hostingPlanName')]"
      },
      "resources": [
          {
              "apiVersion": "2014-06-01",
              "type": "Extensions",
              "name": "MSDeploy",
              "properties": {
                "packageUri": "https://storageaccount.blob.core.windows.net/packages/website.zip",
                "dbType": "None",
                "connectionString": "",
                "setParameters": {
                  "Application Path": "[parameters('siteName')]"
                }
              }
          }
      ]
    }
]

It’s worth noting that ARM only works with V2 Azure resources, meaning V2 VMs and Storage, SQL V12, and so on. Some resources are not yet deployable using ARM such as Azure AD, Azure Service Bus, and Azure Remote App.

Powershell Desired State Configuration

Azure resource manager lets you deploy Azure resources and if you're solely deploying websites or SQL databases that may be enough. However, if you're deploying VMs, be aware that ARM won’t directly configure the VM operating system or install software; for this, you’ll need to use Azure Extensions. There are three Extensions that can be used to configure a VM and install software: Chef, Puppet, and PowerShell Desired State Configuration (DSC). We will focus on DSC for this article, but if you have existing Chef or Puppet deployments you can easily use these standalone or alongside DSC.

Where ARM allowed you to define Azure resources using JSON, DSC defines the configuration of your Windows VM resources using PowerShell. DSC provides hundreds of resources to configure Windows VMs as well as providing a way to write your own custom resources, or use custom scripts in your resources to do exactly what you want. Using the DSC extension then allows you to define the DSC file to be applied to any VM built using your ARM template.

Once you have defined your DSC files, Windows compiles these down to industry standard MOF files, and it's the MOF files that get applied to the VM. This means that you can easily integrate DSC with tools like Puppet and Chef if required and get the benefit of Windows specific resources.

Finally, the DSC engine will continuously apply the DSC configuration you have provided to prevent configuration drift and also apply any changes you make to the template over time. In this way your VM should remain in line with your configuration definition at all times. Windows will automatically test the machine against the configuration file at regular intervals and bring the machine back in-line if required. You can also manually trigger a test with the "Test-DSCConfiguration" command.

Below is an example PowerShell DSC configuration that will install IIS onto a VM, add the .net 4.5 role and then create a website.

configuration IISInstall
{ 
    Import-DscResource -Module xWebAdministration             

    # Install the IIS role 
    WindowsFeature IIS  
    {  
        Ensure          = "Present"  
        Name            = "Web-Server"  
    }  

    # Install the ASP .NET 4.5 role 
    WindowsFeature AspNet45  
    {  
        Ensure          = "Present"  
        Name            = "Web-Asp-Net45"  
    }  

    # Stop the default website 
    xWebsite DefaultSite  
    {  
        Ensure          = "Present"  
        Name            = "Default Web Site"  
        State           = "Stopped"  
        PhysicalPath    = "C:\inetpub\wwwroot"  
        DependsOn       = "[WindowsFeature]IIS"  
    }  

    # Copy the website content 
    File WebContent  
    {  
        Ensure          = "Present"  
        SourcePath      = "C:\Program Files\WindowsPowerShell\Modules\Website" 
        DestinationPath = "C:\inetpub\Website" 
        Recurse         = $true  
        Type            = "Directory"  
        DependsOn       = "[WindowsFeature]AspNet45"  
    }  

    # Create a new website 
    xWebsite WebSite  
    {  
        Ensure          = "Present"  
        Name            = "Website" 
        State           = "Started"  
        PhysicalPath    = "C:\inetpub\Website"  
        DependsOn       = "[File]WebContent"  
    }  
}

Azure Automation

The combination of ARM and DSC is enough to go from zero to a fully configured and ready to use deployment; however triggering and maintaining the deployments is either a manual process, or will require an external orchestration tool. An alternative to this approach is to make use of Azure Automation.

By storing your ARM scripts in Azure Automation you can trigger deployments manually, on a schedule, or allow an external application or process to trigger them using webhooks. Azure Automation could allow you to build a self service way for your users to deploy and tear down their own environments.

Where Azure Automation really comes into its own is the recent update to use DSC and in particular to provide a DSC pull server. A pull server provides an alternative way to get your DSC files to the hosts. Instead of using the ARM script to apply the DSC extension and a particular DSC file to a host, the host registers with the DSC pull server and pulls its configuration from here. By using Azure Automation to act as a pull server you can maintain a centralised repository of DSC scripts which you can easily update and have all of your host pull the changes, rather than updating the scripts on each host. You can also easily see the status of each of your nodes and whether they are in compliance with your desired configuration.

Tips and Tricks

ARM and DSC are not without their issues and idiosyncrasies. Below are some tips when working with these technologies:

• Ensure you keep the version of the DSC Extension you are deploying in your ARM scripts up to date as new versions are released all the time, and older versions of the extension are occasionally disabled automatically. Consider using the "autoUpgradeMinorVersion" flag.

"properties": { 
    "publisher": "Microsoft.Powershell", 
    "type": "DSC", 
    "typeHandlerVersion": "2.8", 
    "autoUpgradeMinorVersion": "true" 
        … 
}

• If you are installing multiple VM extensions using ARM, configure your dependencies so that the extensions install one by one. Installing multiple extensions at the same time can cause errors in the deployment to occur.

• ARM scripts can become large and difficult to navigate. Use Visual Studio and the built in JSON explorer to make navigation between objects and viewing hierarchy easier (requires the Azure SDK).

• Keep your scripts and configurations in version control. This will let you keep a full history and audit trail of the changes made to configurations over time.

Bringing It All Together

We’ve discussed three different technologies in this article: ARM, DSC, and Azure Automation. Hopefully it’s clear now that each of these present part of the solution to an end to end deployment. Used together these solutions allow you build an automated solution to deploying and configuring resources in Azure and keeping these resources in line with your desired configuration throughout their lifetimes.

There is no one way that these tools can be used - they can be combined together to meet your needs. For example, if you're deploying a website with an SQL backend you’ll likely use ARM but never touch DSC. If you're already using tools like Chef and Puppet you may not use ARM, but you might choose to use DSC to give you the benefit of the Windows specific tools that still compile down to MOF files. If you're a System Centre user you may not need the automation and orchestration that Azure Automation provides, but still want to use the deployment tools available in ARM and DSC.

Hopefully this article has given you an insight into how these exciting new features in the Azure platform could help your deployment process become more agile and error free. Using these tools you can focus on "what" you want to deploy, and let PowerShell and Windows take care of “how”. Once we do that we can start reaping of benefits of more reusable, testable, economical and automated deployment processes.

If you're interested in finding out more about these technologies take a look at some of the links in the resources section, or drop me a line on email or Twitter.

Useful Resources

Azure Resource Manger Quick Start Templates - https://azure.microsoft.com/en-gb/documentation/templates/

PowerShell DSC Overview - https://msdn.microsoft.com/en-us/PowerShell/DSC/overview

PowerShell DSC Resources Gallery - https://www.powershellgallery.com/

Azure DS Extension - http://blogs.msdn.com/b/powershell/archive/2014/08/07/introducing-the-azure-powershell-dsc-desired-state-configuration-extension.aspx

Azure Automation and Powershell DSC - https://azure.microsoft.com/en-gb/documentation/articles/automation-dsc-overview/

Pester - PowerhShell Testing Framework - https://github.com/pester/Pester

Day 3 - Design Thinking

Written by: Ben Rockwood (@benr)
Edited by: Shaun Mouton (@sdmouton)

DevOps has many meanings and definitions, spoken and unspoken alike. I have maintained for some time now that DevOps is, above all else, a banner of change which has ushered into the world of IT a variety of ideas, new and old.

The truth of what DevOps really is can be seen in the previous paragraph when I said “ushered into the world of IT.” Once upon a time “Developers” were called “computer programmers” and “Operations” were called “Systems Administrators”, but today those formal titles seem very constricting and inaccurate. Furthermore, both programmers and administrators were both in a department called “Information Technology”, but in most modern companies IT refers only to a small sub-set of technology related activities within an organization, namely those related to servicing employee equipment and infrastructure (laptops, printers, WIFI, etc.)

It is therefore evident today that technology is an integral part of business operation in a new breed of 21st century companies, where the assembly line on a shop floor has been replaced with a continuous integration pipeline accessible to anyone operating from anywhere with internet access. The world has changed and the name ascribed to this transition from back office IT function to core business function is “DevOps”.

DevOps has helped us transition from support function to “business operations.”

Within the DevOps movement have been many so-called “thought leaders.” The function of a thought leader is to seek out and bring into the DevOps movement ideas and methods from other schools of thought. History teaches that great innovation has always come from cross-disciplinary teams working together towards a common goal. Within DevOps we’ve seen an interesting alternative form of this, wherein individuals from within Technology go out and become excited by other disciplines, learn all that they can, and bring back into the community what they have learned.

The large milestone contributions to DevOps have thus far come from the Agile software movement, from the LEAN and Operations Management (Goldratt, Deming, Ohno, Shingo, Shewhart, Taylor, etc) movements, and from Human Factors and System Safety. These three disciplines form the foundation of the body of knowledge from which DevOps draws.

To a lesser degree DevOps has drawn into it knowledge from every discipline which it has happened upon. Emotional Intelligence, Organizational Behavior, Psychology, Philosophy and Epistemology (namely, Pragmatism), Cybernetics, Neuroscience, Architecture, and countless other disciplines are often cited but rarely attributed directly in any significant way. The ideas from these disciplines are simply adopted and become lost within the developing common tapestry of DevOps ideas.

In 2015 we’ve happened upon another discipline which is starting to meld with the DevOps tapestry, that of Design Thinking.

Design Thinking, like DevOps itself, is somewhat controversial as to whether or not it is in fact something new or simply a fad. The term was coined in 1969 by Herbert Simon in his book The Sciences of the Artificial and then made famous in 1987 by Peter Rowe’s book Design Thinking. Through the 1990’s several articles and books were written which broadened the scope of Design Thinking from a way to create new things to a method of approaching complex problems. In 1991 David Kelley founded the company IDEO, an international design and consulting firm in Palo Alto, California, which has subsequently become the poster child for innovation and the applications of Design Thinking.

The rise in popularity of design thinking is due to the belief that the way to succeed in business today is by means of innovation and design. The poster children for success in the 21st century are businesses like Apple, Amazon, and Google which are now the most valuable companies in the world, and it is innovation that is believed to put them there. I say “believed” because when other businesses look at them it is “innovation” that they single out as the thing they are missing and try to emulate. The problem however is that innovation isn’t something you can simply start doing, its not so easy to copy.

Like DevOps, design thinking starts with culture. It’s more of a worldview than a process, but because it does have a process it is only natural that non-innovative companies will take it, try to implement it, and hope for the same dramatic results.

IDEO’s David Kelley and Tim Brown are all too happy to share the design thinking message with the world, and business schools have jumped on the bandwagon, especially Roger Martin, Dean of University of Toronto’s Rotman Business School, which is closely aligned with his alma mater Harvard Business School. Tim Brown wrote Change by Design in 2009. That same year, Roger Martin wrote The Design of Business. David Kelly, along with brother Tom Kelley, wrote Creative Confidence in 2013, although Tom Kelley (who also works at IDEO) wrote The Art of Innovation back in 2001, ahead of the trend.

The design thinking process is straightforward and will look familiar to anyone in the Agile or DevOps communities:

1. Empathize: Develop a deep understanding of the challenge
2. Define: Clearly Articulate the problem you want to solve
3. Ideate: Brainstorm potential solutions, Select & Develop your solution
4. Prototype: Design a prototype (or series of prototypes) to test all or part of your solution
5. Test: Engage in a continuous short-cycle innovation process to continually improve your design

The primary contribution of design thinking to the world is an emphasis on step 1 “Empathize”. This means that design thinking starts with a human centered focus and attempts to understand the problem from his or her perspective. Any problem address that doesn’t have human beings at the center is not a design focused solution. When a problem has multiple stakeholders, that is, humans who are affected differently by a single situation or problem, each of their perspectives is sought out to help arrive at a solution which addresses them all, which is a form of systems thinking.

Once a problem is understood, it is defined, typically via a “design brief”. The brief is a document which outlines the actual problem, not a proposed solution, along with context and analysis discovered in the empathize stage, so that a team can be assembled to begin the ideation process.

Ideation, Prototype, and Test form a cycle. The problem is considered, ideas are brainstormed and examined, and eventually those ideas are synthesized into one or more prototypes. Those prototypes are experiments which are created and tested. Based on those results the cycle can begin another iteration.

Several elements should sound familiar to the DevOps and Agile communities! User Stories are a way to empathize with the customer and define problems from a human perspective. Scrum implemented properly looks a lot like the design thinking process. The LEAN Plan-Do-Check-Act (PDCA), especially its Toyota created implementation as an A3 Report, are strikingly similar to the design thinking process as a whole. Moreover, the Agile and DevOps philosophies at large seem to naturally resonate with the design thinking methodology.

Despite all the similarities, there are 3 things which we need to absorb from the Design Thinking community that we often neglect at our peril.

The first is that humans must always be at the center of creation. Systems, however complex they may be, ultimately serve one or more humans. When we create systems which are optimal for their own sake we create an environment which turns humans into machines. Humans should create machines, not serve them.

The second is that there is never only one solution, we must explore and try as many as possible. The LEAN Startup of Eric Ries (which is really from his mentor Steven Blank) introduced the concept of Minimal Viable Product (MVP). The intention of an MVP is to invest the absolute minimum required effort to produce something to sell a potential consumer. If the customer doesn’t want to buy what you’re selling them, you need to know as soon as possible and change your approach, or pivot, towards what they will buy. Today, however, companies invest months or year on a “beta” which they then call an “MVP”, when in fact they left the MVP stage after the first week of effort invested. Prototyping enables quick experimentation and evaluation to iterate closer and closer to the solution your customer (whoever the customer is) wants and that means simple, fast, and iterative.

Finally, and perhaps most importantly, in order to create new and innovative solutions we must, by definition, stop acting like integrators who assemble a variety of available piecemeal solutions together to simply arrive at a solution as quickly as possible. We must step back from the problem, allow ourselves to empathize with the stakeholders and realize that the existing situation is a result of its place within a system which supports it. We must understand the entire system, the problem in its full context, and then define the problem within that systems context. Only then can we ideate true “break out” solutions which change the game, which constitutes true innovation.

Design Thinking is a powerful focusing lens and set of methodologies which can help reinforce our existing practices and improve upon them. It is a trend that already holds a central place in the world around us and one that we can learn and embrace.

2015 marks an important evolution in the DevOps movement, the adoption of design thinking principles. Several of us in the community have independently come to an appreciation of it. Jeff Sussna gave what I believe to be the best talk of 2015: From Design Thinking to DevOps and Back Again: Unifying Design & Operations (https://vimeo.com/129939230) and, moreover, LEAN UX 2015 showed itself to be the most important conference of the year by embracing the design thinking approach into a wide variety of tech disciplines, beyond just UX. This is a trend I hope to see expand greatly within the coming year!

Day 2 - Examining Tools With a Devops Lens

Written by: Jennifer Davis (@sigje)
Edited by: Greg Poirier (@grepory)

Over the last few years I have had the opportunity to attend a variety of conferences, meeting people and hearing diverse stories about work environments and challenges. It is fascinating to hear the variety of explanations about devops and its ongoing impact on the industry and the workplace. As I worked to gain a better appreciation of others' experiences, I realized the challenge of misinformation about devops. How do we expect individuals to understand and choose devops when we have so many competing themes?

Two big themes I have heard:

• Devops is culture not tools.
• Devops is automation.

In this article, I will tackle one aspect of the challenge focusing on examining tools within the industry with a devops lens.

Devops is culture not tools.

What is culture? Culture is the totality of learned, socially transmitted customs, knowledge, objects, and behaviors. It includes the ideas, values, customs, and artifacts of a group of people. Culture encompasses everything about how we live and work, and how life and work evolve to meet the challenges we have in our environments. It gives meaning to social, political, economic, aesthetic, religious norms, and modes of organization that distinguishes us from others.

"We become what we behold, we shape our tools and then our tools shape us."

Father John Culkin

What are tools? Tools are a vital component of our identity, how we interact with the world around us, how we use and conserve energy, and how we buffer ourselves from harm. Tools shape our thoughts and behaviors including our transparency, level of exertion over our environment, and conflict response. Tools are a cultural artifact.

Consider the introduction of automobiles and the changes to city landscapes that came about because of them; new cities have wider streets to accommodate vehicles with less care taken to provide pedestrians with safe walking. Automobiles impacted how we work and where we work. As commute times have increased in densely populated regions, the value of owning a car has decreased and the landscape will transform again.

Prior to the introduction of devops, an organization might incentivize and reward teams based on each team's particular priorities; developers strove to build features quickly, operations stabilized systems, security minimized risk and controlled access, and so on across the organization. When the bonus for the VP of Operations is driven by uptime, system administrators will be incentivized to ensure stability and availability. When the bonus for the VP of Development is driven by meeting deadlines, developers will be incentivized to deliver features. This leads to a fundamental conflict in deciding highest priority between individuals trying to work together across organizational boundaries.

Organizations have created islands of culture based on their existing individual and collective values. For example, the roles and responsibilities of an operations engineer at one organization may significantly differ at another. Consequentially, operations engineers at different companies often have substantially disparate skill sets. Over time, the roles and concerns within an organization change leading to fractured capabilities even within a single organization.

As organizations grow larger, each team's individual focus narrows further. Each team adopts a specific tool that meets their requirements, leading to multiple tools that overlap in purpose. This proliferation of tools both hinders transparency within the organization and stifles collaboration.

For example, bug tracking within development teams led to the development of tools like JIRA and bugzilla. Ticket tracking within operations teams led to development of tools like Request Tracker(RT). Bugs and tickets are both issues. While a single issues tracking tool can be selected at the organizational level, the tool can impact the cognitive load of teams unequally as a team tries to adopt a tool that isn't specific to their domain.

The problem of tool proliferation continues today with the belief that a specific tool might solve all the problems an organization may have. When people say Devops is culture not tools, it is in part to focus on solving the organizational issues that can not be solved with a change in technology alone.

What is Devops?

Devops is an ideology that seeks to change how individuals think about work, value the diversity of work done, develop deliberate acceleration of business value, and measure the effect of social and technical change.

In Effective Devops, Ryn Daniels and I introduced the 5 pillars of devops: Collaboration, Hiring, Affinity, Tools, and Scale.

These 5 pillars form the foundation for effective devops and the devops compact, the continuous building of a shared mutual understanding within a team, organization, and across the industry.

With the devops compact, we commit to working together, communicate our intentions and any issues, and adjust our work in order to work towards shared organizational goals. We answer questions like: what is the value of what we are we trying to do, how are we going to do it, and what exactly are we doing? We don't just decide to do devops, and then we’re done. The nature of doing devops, means that we are commiting to adjusting to change.

Devops is culture which includes tools and their use. Tools are not devops. How tools are used, and the ease with which they can be used impacts the acceptance and proliferation of specific aspects of culture. When we talk about devops tools, we mean the tools and the manner of their use--not fundamental characteristics of the tools themselves.

Devops tools stress "We" over "Me"; they allow teams and organizations to build mutual understanding to get work done. Your choice of tools is a choice in a common language. Is this language one that benefits your organization as a whole or merely a subset of specific teams? At times, due to the lack of availability of an equally balanced tool, a choice must be made that has a higher cognitive cost to one team over another. Be aware of the cost and empathetic to the teams impacted.

The devops culture that we embrace is one of collaboration across teams, across organizations, and across industries. When developing solutions, I think about my team instead of thinking about the easiest thing for me to do now. This sometimes means adjusting my expectations for the good of the organization and choosing something a little more difficult than my current working structure.

Tool Examination

Let's examine a few tools with the devops lens. First we'll look at version control systems and then infrastructure automation. Please note that this examination of tooling is not meant as a judgement or advocacy of a specific tool in your environment. Without understanding your organization's culture, noone can tell you the right tool to choose.

Version control systems record changes to a set of files over time. CollabNet founded the Subversion project in 2000 as an open source software versioning and revision control system that was architected to be compatible with the widely used Concurrent Versions System (CVS). Subversion 1.0 (svn) was released in February of 2004. Technology and habits at the time dictated svn's use and features. Core to svn's architecture is the concept of a centralized repository. This central repository allowed users to control who was and was not allowed to commit changes.

A year later in 2005, Git was released. It’s also an open source software version control system with a focus on decentralized revision control, speed, data integrity, and support for distributed nonlinear workflows. This gives every developer full local control. While you can adopt a centralized workflow and establish a "central" repository, the processes can be flexible allowing you to use technology as you wish rather than having it defined for you. While the ramp up time may be a little longer, the functionality allows for quicker organizational changes.

Andy Peatling shared the WordPress technical refresh in migrating from svn to git in 2015, and the desirable changed behaviors that included:

• improved developer communication through code reviews and hangouts
• improved development practices of reviewing code prior to committing to the repository because of the availability of the pull request with git
• increased collaboration between designers and developers on a daily basis
• greater feedback on individual work,
• customized workflows

The technical merits between git and svn aside, significant value is derived from increased cross-team collaboration.

In many organizations, system configuration is a manual process. Individuals document the process and upgrade with a checklist. A missed step can lead to systems in an unknown state requiring considerable effort to recover.

Chef is an infrastructure automation system. Infrastructure automation is creating systems that reduce the burden on people to manage services and increase the quality, accuracy, and precision of a service to the consumers of a service.

When Adam Jacob was developing Chef software, he was trying to create a solution that could work across different organizations. Chef was built to provide abstractions for configuration and management, creating a language allowing individuals to define its infrastructure and policies with code.

Trying to create a language that allows for the nuanced views of developers, system administrators, security operations, and quality assurance engineers is difficult. With Chef, rather than reusing terminology that shows preference to one role over another, we have new terminology including resources and recipes.

Resources make up the basic building blocks of our infrastructure. We can use resources as provided by Chef core libraries, use resources provided by the larger Chef community, or we can build and customize our own.

Recipes describe a specific piece of an application that we want to have running on a system. It's the ordered set of resources and potentially additional procedural code. Just as with a recipe for baking chocolate chip or oatmeal cookies, the recipe will be specific to what we want to create.

In this simple webserver.rb recipe from the Learn Chef website, we use 3 resources: package, service, and file. These resources describe everything needed to stand up an apache web server, have it run at boot time, and have it serve a simple "hello world" page.

More complex tasks like opening specific ports in a firewall or deploying Hadoop can be abstracted into a common language that allows for comprehension across teams. Because this code is stored in your standard version control, you can take advantage of the same ease of use and transparency that the rest of your source code is afforded.

Devops is not limited to infrastructure. A lot of focus is on tools that improve collaboration between development and operations teams as those teams are where the devops movement originated. As other teams begin adopting devops methodology, we'll see tools that support the devops compact between new teams. For example, Chef has released Chef Delivery this year. Chef Delivery simplifies organizational complexity by creating a common language to describe the process of promoting products between teams. This common language allows us to codify "delivery as code".

Devops is Automation

When people say Devops is Automation, it is due in part because many tools in devops, while codifying understanding to bridge the chasm between teams and increase velocity, have resulted in automation for repetitive tasks. Automation is a result of improved technology focused on repeatable tasks.

While reviewing the aviation industry in 1977, the House Committee on Science and Technology identified cockpit automation as a leading safety concern. Studies of the aviation field discovered that while pilots could still fly planes, the automation in use was causing an atrophy to critical thinking skills. Pilots were losing the ability to track position without the use of a map display, decide what should be done next, or recognize instrument system failures. This is a warning for us as we implement automation within our environments. Tools change our behavior and the way that we think.

In July 2013, Asiana Airlines flight 214 struck a seawall at San Francisco International Airport, resulting in three fatalities. During the investigation, the National Transportation Safety Board (NTSB) identified a number of issues, among them that there was insufficient monitoring of airspeed due to an overreliance on automated systems that they didn't fully understand.

“In their efforts to compensate for the unreliability of human performance, the designers of automated control systems have unwittingly created opportunities for new error types that can be even more serious than those they were seeking to avoid.” James Reason, Managing the Risks of Organizational Accidents

Automation is critical to us as systems become more complex and organizations become interdependent due to shared services. Without shared mutual context or concern for human needs, automation creates unknown additional risk.

Devops enabled automation may make work faster, but more importantly it has increased transparency, collaboration, and understanding.

In summary, devops is culture which includes tools and their use. Tools help us progress from manual to continuous processes. Effective devops tools enable automation without isolating humans from the automation process so that the mutual understanding occurs across time, ensuring that the "we" of tomorrow understands the "we" of today.

What Next?

Find a local DevOpsDays, CoffeeOps, or other cross-functional meetups that facilitates people coming together to build strong foundations of common understanding.

The devops community calendar also lists upcoming conferences and events.

Study

• Github git tutorial
• Atlassian Git tutorial
• History of Chef
• Learn Chef

Watch

• DevOpsDays Silicon Valley 2015 Videos
• Tools for Effective Change Webinar

Listen

• Arrested DevOps
• Foodfight Show
• DevOps Cafe

Read

• devops Weekly - weekly newsletter of curated devops news from Gareth Rushgrove
• Comparison of Issue Tracking Systems
• Comparison of Version Control Software
• NTSB report for Asiana Airlines flight 214
• Effective Devops

Share

I'd love to hear your feedback! What tools are you using in your environment and how do they impact organizational culture?

Tweet at @sigje with hashtag #sysadvent, email me at sparklydevops@chef.io, or add a comment to the article.

Thank yous

Many thanks to Greg Poirier for being my Sysadvent Editor. Additional thanks to Nathen Harvey, Peter Nealon, Amye Scavarda, and H. Waldo Grunenwald for peer review and aditional edits.

• Image Credits
  • Young Girl Tilting Camera Up. Licensed from Kenneth Wiggins on https://blackstock.co/collection/young-girl-tilting-camera-up/
  • Light Sky Silo Windows Sunlight Rays from Public Domain Archive

Day 1 - Using Automation to build an OpenStack Cloud

Written by: JJ Asghar, @jjasghar

Edited by: Klynton Jessup, @klyntonj

I wrote this as a narrative around what I hope a typical engineer would experience trying to resolve an issue. This story, while fiction, is taken from personal experiences and inspired by what I hope would happen. I hope you enjoy this.

Like normal, I came to my stand-up unexcited for the normal grind. My boss Steve came in, sat down at the conference table, and put his notebook down. Yes, stand-up happened in a conference room and yes we actually sat at the conference room table, so, honestly, I never really understood why we called it a stand-up. I guess it was a “rebranding” of “Daily Status” or maybe it was a hold over from those days we tried to do “agile.” Who knows?

Anyway, Steve opened his notebook and looked around at my team. “So, we have a problem. The local development for our cookbooks is great but we need to start testing on multiple platforms. There’s a chance we might be spinning up a new application and it only runs on CentOS.” (We’re a Ubuntu shop.) There were some sighs and groans as we looked around at each other.

“Today y’all have your normal responsibilities but we need to think of a way to parallelize our cookbook development. I dunno if you’ve ever tried it but running kitchen test -p 2 on your laptops brings everything to a grinding halt. Let’s skip stand-up and spend the first half of the day doing some research and let’s try and come up with some ideas. We’ll get back together after lunch.” With that Steve closed his notebook, got up, and walked out of the room.

“Interesting,” I said under my breath. “I guess it’s time to start Googling.”

I walked back over to my laptop and typed in “parallelize cookbook development” in Google. Wow, there was nothing about test-kitchen on there, until I got to http://leopard.in.ua/2013/12/01/chef-and-tdd/ on the second page! It’s 2015, so this post from 2013 has to be out of date. Right? But didn’t Steve mention the -p in stand-up? I read through the page and found out that there was a kitchen test --parallel that must be what he was talking about. Sweet. OK, he isn’t confused and just saying words again.

I hopped over to https://github.com/test-kitchen/ and noticed all the drivers for test-kitchen. There were a ton, ranging from AWS, to HyperV, to OpenStack, and Digital Ocean (DO). This is great, I can run test-kitchen on any cloud I want. I started to play around with the different options and settled on https://github.com/test-kitchen/kitchen-digitalocean as my test version. I’ve always enjoyed using Digital Ocean to do my development and it seemed reasonable. I opened my .kitchen.yml and looked at the configuration.

driver:
  name: vagrant
  network:
    - ["forwarded_port", {guest: 80, host: 8080, auto_correct: true}]
  customize:
    cpus: 4
    memory: 8096

I made the changes to the driver name and added in the Droplet size I wanted:

driver:
  name: digitalocean
driver_config:
  size: 8gb

And then installed the gem:

gem install kitchen-digitalocean
export DIGITALOCEAN_ACCESS_TOKEN="THIS_ISNT_MY_REAL_TOKEN"
export DIGITALOCEAN_SSH_KEY_IDS="8675, 30900"

Then I ran test-kitchen. Nice! It worked. This is great. I spun up both of my test boxes on Digital Ocean, one Ubuntu 14.04 and one CentOS 7 at the same time and verified them both. I got up and walked over to Steve’s office. “Hey Steve, I think I got an answer for you from stand-up,” I said.

“Oh yeah? That was fast,” he said looking up from his laptop. “What is it?”

“I hooked up test-kitchen to Digital Ocean and ran a test command on both systems at the same time,” I said, with confidence.

“Ha! That’s awesome. I had no idea you could use other drivers with test-kitchen, I thought it was all local development.”

“Yeah I learned that after looking at the GitHub test-kitchen org, turns out it was pretty easy to set up.”

“Hang on, Digital Ocean costs money though right?”

“Yeah, it’s only pennies though, and I charged my own personal account.”

“Ah well if it’s ‘only pennies’ I guess you won’t need to expense it.”

“Ouch, fine. I guess I deserve that. So you think it might be too expensive to run our test suite for every commit?”

“You read my mind. We need something like DO but local. How about OpenStack?”

“HAHAHAAHAHAAHAHAHAHAHAH” I actually started crying from the tears of laughter. “Yeah right, I don’t have enough time to set up an OpenStack cloud. OpenStack has only gotten worse since they started the project.”

Steve took a moment to let me compose myself, then said plainly, “You should give it a shot, I heard JJ, the OpenStack Chef guy, mention something called an ‘OpenStack-model-t.’ It’s a project Chef was working on to help people build basic OpenStack clouds. Any chance there’s a kitchen-openstack driver for it like there was for Digital Ocean?”

“Actually, yeah there is. Ah, OK I see where you’re going with this, I’ll report back what I find out about these two projects.” I turned around and walked out of his office.

I sat down at my laptop, opened up Chrome, and typed in: openstack-model-t. The first hit was: https://github.com/chef-partners/openstack-model-t. One of the first things that caught my eye was: “The Customer Can Have Any Color He Wants So Long As It’s Black”. Henry Ford. Funny, real funny Chef. I started looking around. It seemed that the cookbook was pretty straight forward. There was even a .kithen.yml file in the repo, so I figured I’d checkout the repo and give it a shot.

cd ~
mkdir openstack-stuff
cd openstack-stuff
git clone https://github.com/chef-partners/openstack-model-t.git
cd openstack-model-t
chef exec kitchen verify

My laptop fans started to spin and my MacBookPro started to heat up, yep, I was building an OpenStack cloud on my laptop. After about 20 minutes, I came back to my laptop and I saw:

         Process "nova-scheduler"
           should be running
           user
             should eq "nova"
         Process "neutron-l3-agent"
           should be running
           user
             should eq "neutron"
         Process "neutron-dhcp-agent"
           should be running
           user
             should eq "neutron"
         Process "neutron-metadata-agent"
           should be running
           user
             should eq "neutron"
         Process "neutron-linuxbridge-agent"
           should be running
           user
             should eq "neutron"

       Finished in 1.52 seconds (files took 0.41367 seconds to load)
       67 examples, 0 failures

       Finished verifying <default-ubuntu-1404> (0m18.47s).
-----> Kitchen is finished. (16m24.13s)

Wow, a fully tested and verified All-in-One OpenStack cloud in one command! Let’s see if I can spin up a VM. Reading the README, it seems I have to run a script when I first ssh into the box to make sure I have an initial network and “CirrOS” image on the machine. OK, so be it.

chef exec kitchen login
sudo su -
bash build_neutron_networks.sh

Now the README tells me I can look at the URL in my web browser, with https://127.0.0.1:8443/horizon and I should see the OpenStack login. I opened up Chrome again and typed it in. Sweet, it worked. Using the demo username and password of mypass, I was able to login without a hitch. I clicked the Instance button on the left-hand side and then “Launch Instance”. Heavens to Betsy, it actually worked!

I looked around, I didn’t believe it. A successful OpenStack build out of the box? What kind of black magic was this? This is bonkers. I stood up from my laptop and took a walk. This opened up a huge opportunity for my team and I needed to clear my head.

As I walked around my office I saw one of our IT team members carrying a new laptop that he had provisioned for someone on another team. He was going the same direction I was, so I figured I’d ask, “Hey Billy, quick question what do you do with the old desktops that these laptops are going to replace?”

I guess I caught him off guard because he turned and looked at me with confused and concerned eyes, “Oh, hey, honestly, nothing. They have already depreciated in value so they just sit in the IT room until we get Goodwill to come ’round and we write them off as a donation. Nothing too exciting.”

“Interesting, any chance I can get…three of them, I’d like to try something out.”

“Sure, no problem, come ’round in a bit and I’ll get you what you need.”

“Awesome, see you then.” And I continued walking.

About twenty or thirty minutes later I walked up to the IT room and Billy was looking at Reddit. No real surprise there, 60% of IT work is done or is linked to from Reddit and yes you can quote me on that. “Hey Billy, I’m here, can you hook me up?”

“Sure, no problem, take what you need; just write it down on that clipboard.”

“Can’t I email it to you?”

“Meh, you could, but I’d prefer to see you write down what you take when you take it.”

“Fair enough,” I said, as I picked up one of the desktops. “Any chance you have a spare switch or two?”

“Probably, just be sure to put it down on that clipboard if you find one.”

“Cool, thanks again.”

“No problem”, he said as he turned back to looking at his laptop.

I pulled together what I needed and took it all back to my desk. The README for the model-t had a controller node that needed 3 NICs in it and the compute nodes only needed 2 if I wasn’t going to use a storage network. I thought about it for a few moments and realized that nope, I don’t need a storage network for running a test-kitchen OpenStack cloud, so 2 NICs would be fine.

I powered on the controller node, it had Windows 7 on it, which meant I had to re-image these machines. I downloaded Ubuntu 14.04, created a USB boot disk, and started the installation on the machines. I tried doing it in parallel but ended up confusing myself; so I did it serially. I rebooted the controller node and started the installation. I did a basic install, naming the machine Controller, original I know, and connected it to my lab network. I was able to ping 8.8.8.8 with only my management network plugged in so I felt like a was making some progress. I repeated the process with the two Compute machines naming them Compute1 and, you guessed it, Compute2 and then decided to break for lunch. Over lunch I talked to some of my coworkers about what I was doing and all-in-all there was a pretty positive response. I did get a couple giggles and shakes of their heads about OpenStack, but I expected that.

I walked back to my desk, looked at the beginning of what I was hoping to be a real OpenStack cloud and was pretty proud of my work. I remembered then that Steve wanted to have a sync up after lunch, so I headed over to the dreaded stand-up conference room. As I walked in Steve was just starting.

“So it seems we have a couple options on the table. Most of you figured out that you could run test-kitchen with different drivers giving you access to more compute resources. That’s good. Some of you didn’t come to me with anything so I have to assume you either got caught up with something else, or didn’t bother looking.”

After that inspiring talk, I spoke up, “Hey Steve, I’ve made some pretty impressive progress with the OpenStack-model-t cookbook, I’m going to go heads down on it for the rest of the day to see if I can get this done by EOB.”

Steve looked at me, smiled, then looked at the rest of the room and said, “See that’s what I want, someone to run with my assignments. OK, let’s get back to work and let’s see where we are at stand-up tomorrow.” We all started filing out of the conference room, all of us realizing at that moment how much of a waste of our time our sync meetings were.

I sat down in front of my laptop and started to think of what I had to do next. I put a small plan together:

1. Get a hosted Chef instance, the 5 free nodes they give me ought to be enough to get this proof of concept built.
2. Get the model-t cookbook up on the hosted instance.
3. Figure out what I need in the run_list for each of the machines.
4. Converge and see my cloud come to life.

First is always first, right? So I went to https://manage.chef.io and got myself an instance of hosted Chef. It was pretty easy I just created a new org, baller-model-t and then pulled down the getting_started kit. I did a chef exec knife status to confirm it was working and it was. Awesome, step 1 complete. Second, I uploaded the openstack-model-t cookbook to my hosted instance. It complained about dependencies because I had forgotten to do the Berkshelf stuff. So I did the following:

cd openstack-model-t
chef berks install
chef berks upload
cd ..
chef exec knife cookbook upload openstack-model-t -o .

Sweet, success. OK, as it looked in the cookbook the All-in-One test was sent from the default.rb recipe. That’s good, that’s my controller node. Now the question is what’s in my compute node run_list? Well it looks like compute_node.rb is the wrapper for just a compute node. Great, I’m almost done. Now to get the converge to happen. I still need to get these boxes to check into the hosted Chef instance, so I decided to use knife bootstrap to do it in one shot. These were the commands I used:

chef exec knife bootstrap controller -x ubuntu --sudo -r 'openstack-model-t::default'
chef exec knife bootstrap compute1 -x ubuntu --sudo -r 'openstack-model-t::compute_node'
chef exec knife bootstrap compute2 -x ubuntu --sudo -r 'openstack-model-t::compute_node'

I went to https://controller/horizon and I couldn’t believe what I saw. I logged in with admin/mypass and I saw 3 hypervisors and an empty OpenStack cloud ready to go. I ssh’d into the controller node, ran the bash build_neutron_networks.sh, and saw it come to life. (NOTE FROM MY FUTURE SELF: If I remember correctly, I had to change some of the floating-ip options around on my second build of this, but still, this was amazing.)

I spun up a CirrOS image, then a second, and a third. I ssh’d into each of them and make sure I could ping 8.8.8.8. It worked! I had a running multi-node horizontally scalable cloud on my desk. I looked at the clock and it was only about an hour ‘til quitting time. I searched for some OpenStack cloud images and found Ubuntu’s and CentOS’ and injected them into my cloud. I leaned back for a moment and thought about what I had accomplished. I had to smile. As someone who has always been interested but scared to try building out OpenStack this seemed like dream come true.

For a couple moments, I thought about going home, then I realized that I could finish the project off if I just got kitchen-openstack running. So I went through the same steps that I did with Digital Ocean with the kitchen-openstack driver. I ran my parallel tests and successfully spun up a test-kitchen run.

I couldn’t be more proud of my accomplishments for today. For the first time in a while I’m going to be excited for stand-up tomorrow when I can show this off to everyone.