Day 18 - Deployments done the Delivery way

Written by: Christopher Webber (@cwebber)
Edited by: Ted Young (@jitterted)

This year has been all about Delivery for me, starting with getting https://www.chef.io deployed on stage at ChefConf using Delivery. It has been a blast moving services into Delivery and using Delivery to build new ones.

What Even is Delivery?

In the simplest of terms, Delivery is tool for enabling continuous delivery. It has been shaped over many years of experience working with folks all across the industry in building their pipelines. For me, it is an opinionated build and deployment pipeline. Explaining why things are the way they are is a bit outside of the scope of this post. What follows is a brief overview of the way I view the world.

Phases and Stages

Delivery is made up of a set of stages: Verify, Build, Acceptance, Union, Rehearsal, Delivered. There are manual approval steps between the Verify and Build stages, and the Acceptance and Union stages. Each stage is made up of a series of phases where actual tasks are executed.

Below is a list of the stages, and the phases that they execute.

• Verify: Before another developer reviews the code, verify it is worthy of being viewed by a human.
  • Unit: Unit test the code that you are deploying. For a cookbook, this is probably ChefSpec, for a Rails app, it might be RSpec or minitest.
  • Lint: This is a test of whether your are properly formatting your code and following best practices. For Ruby apps, you probably will want to run Rubocop.
  • Syntax: Is it parse-able? Just like we do a configtest before restarting nginx or apache, it is useful to do the same with our code.
• Build: Now that code review is done and we have merged to master, let's build some artifacts (cookbooks, packages, images, etc.)
  • Unit: Same as before, but now on an integrated codebase (we merge the code to master during the manual approval step between Verify and Build).
  • Lint: Same as before, but now on an integrated codebase.
  • Syntax: Same as before, but now on an integrated codebase.
  • Quality: This is where you might fail a build it if doesn't have the right amount of code coverage, etc. You are looking to test that the code meets a quality standard of some sort.
  • Security: Test the code for security. In Rails, running Brakeman along with bundler-audit is a great place to start.
  • Build: Produce artifacts that we can promote through the process. This may be a cookbook, a software package, or even a system image.
• Acceptance: Setup the artifact(s) in an environment where we can verify that they are ready to go to production. We have a manual step after this to give someone a chance to poke around and make sure things work well.
  • Provision: What this means for your environment may vary, but I usually use it to stand up the instances I am going to deploy onto and any other supporting pieces, such as ELBs, RDS Instances, Elasticache Instances, etc.
  • Deploy: In most cases, this is a matter of, run the cookbook associated with the service.
  • Smoke: Does it work? For most web services, it is as simple as making sure you get a 200 OK from a healthcheck endpoint to prove, yup, it started. These tests should be super lightweight to provide fast feedbackin case it fails, so we don't waste time doing Functional tests.
  • Functional: This is where we ensure it functions correctly. Whether that is testing a bunch of endpoints, running selenium scripts, or pointing metasploit at the instances, you want to validate that the system is functional.
• Union: Do the upstream services still work? After we do a pass on the service we are deploying, we go and re-run the phases in the union stage for the projects that have declared a dependency on this project.
  • All phases are the same as in Acceptance.
• Rehearsal: Ensure that we can do the deploy one more time cleanly.
  • All phases are the same as in Acceptance.
• Delivered: Actually build out the "production" service.
  • All phases are the same as in Acceptance.

As you may have noticed, most of the phases are executed in more than one stage, allowing us to ensure that the state of the world is good. For example, all of the phases that run in Verify also run in Build to validate that things are still good. And in Acceptance, Union, Rehearsal, and Delivered, each stage runs the same set of phases to build each environment the same way.

Ship it!

So what does this actually look like? For most services, I see it break out into three pieces:

1. The application: This is the actual service you are going to run. It is usually the base of the repo.
2. The deploy cookbook: A cookbook that lives under cookbooks that you run on the node on which the service is running.
3. The build cookbook: A cookbook that lives at .delivery/build-cookbook that handles all of the moving parts that make the service go.

Most of us are familiar with the first two. The application is the actual thing. If you are a Ruby shop, it is probably a Rails or Sinatra app. If you are a Java shop, it might be a Spring app. Whatever it is, it is the actual thing you are deploying. The deploy cookbook is the configuration management code that makes the node do the thing. If you are deploying a Rails app, it probably sets up nginx, adds some users, and spins the application up using Unicorn.

The Build Cookbook

I want to focus on the third item for a bit. The build cookbook is what Delivery, using the delivery-cli, actually runs. Each phase is represented by a recipe. So there are unit, lint, syntax, provision, etc., recipes in this cookbook. There is also a recipe called default, which is run as root, and runs at the beginning of each phase. Once that finishes, the actual phase recipe is executed with non-root privileges. The build cookbook provides the directed orchestration I have always wanted: I can stand up a database, run the data import, and then, only if that succeeds, start up the app instances. In the case of omnitruck, we make sure that the instances have everything they need, like a load balancer and CDN service before we deploy the code.

The Shared Repo

Since all three pieces, the application, and build and deploy cookbooks, are all in one repo, changes to any aspect of the application can easily be found. The coolest thing is that we now tie all changes to the service to a single commit history. This means, if we make a change to the app and a corresponding config change is needed in nginx, we see it all together. It also means that all changes to the app are tracked in one place. Whether we are tracking that a new route was added to the app or that a new header was added via the load balancer, all of the changes are wrapped up neatly in a single log of commits.

Deploying Omnitruck

Chef runs a service called omnitruck that provides information about packages used by chef-client and other tools. The application follows the pattern I outlined above. You can visit the omnitruck repo and browse through the code. Here is a high level overview of what it looks like to ship omnitruck.

1. The process starts with someone creating a change, automatically kicking off the verify stage. If it passes, we review the code and approve the change.
2. From there it heads to build and acceptance. In the build stage, we get a set of deployable artifacts. For omnitruck, it is the deploy cookbook being published to the chef-server and the source code being neatly packaged into a tarball.
3. The fun really begins in the acceptance stage where we start standing stuff up. We provision an ELB, a set of EC2 instances, some CDN config, and some DNS entries.
4. Still in the acceptance stage, we next use the deploy cookbook, which comes from the chef-server, to deploy omnitruck to the EC2 instances. If the chef-client run completes successfully on the EC2 instances, we flush the cache on the CDN.
5. Smoke and functional tests then run to ensure that we are good to go.
6. In union, we do it all over again, except, that we rerun the Union phase of each of the consumer projects, which are other projects that have declared a dependency on the service we are shipping. For example, while you can't see it in the omnitruck repo, there is a project called chef-web-ocfrontend which defines the nginx instances that support www.opscode.com. That service depends on omnitruck so we verify that it didn't get broken in this process.
7. From there, we move on to the rehearsal stage and the delivered stage which makes the project live.

Conclusion

As I watch Delivery mature, I am amazed at how awesome the workflow has become. While the product Delivery is closed source, the delivery-cli, which handles the actual running of code is freely available for download.