Day 14 - Terraform Deployment Strategy

Written by: Jon Brouse (@jonbrouse)
Edited by: Kerim Satirli (@ksatirli)

Introduction

HashiCorp’s infrastructure management tool, Terraform, is no doubt very flexible and powerful. The question is, how do we write Terraform code and construct our infrastructure in a reproducible fashion that makes sense? How can we keep code DRY, segment state, and reduce the risk of making changes to our service/stack/infrastructure?

This post describes a design pattern to help answer the previous questions. This article is divided into two sections, with the first section describing and defining the design pattern with a Deployment Example. The second part uses a multi-repository GitHub organization to create a Real World Example of the design pattern.

This post assumes you understand or are familiar with AWS and basic Terraform concepts such as CLI Commands, Providers, AWS Provider, Remote State, Remote State Data Sources, and Modules.

Modules

Essentially, any directory with one or more .tf files can be used as or considered a Terraform module. I am going to be creating a couple of module types and giving them names for reference. The first module type constructs all of the resources a service will need to be operational such as EC2 instances, S3 bucket, etc. The remaining module types will instantiate the aforementioned module type.

The first module type is a service module. A service module can be thought of as a reuseable library that deploys a single service’s infrastructure. Service modules are the brains and contain the logic to create Terraform resources. They are the “how” we build our infrastructure.

The other module types are environment modules. We will run our Terraform commands within this module type. The environment modules all live within a single repository, as compared to service modules, which live in individual repositories or alongside the code of the service they build infrastructure for. This is “where” our infrastructure will be built.

Deployment Example

I am going to start by describing how we would deploy a service and then deconstruct the concepts as we move through the deployment.

Running Terraform

As mentioned earlier, the environments repository is where we actually run the Terraform command to instantiate a service module. I’ve written a Bash wrapper to manage the service’s remote state configuration and ensure we always have to latest modules.

So instead of running terraform apply we will run ./remote.sh apply

The apply argument will set and get our remote state configuration, run a terraform get -update and then run terraform apply

Environment Module Example

Directory Structure

The environment module’s respository contains a strict directory hierarchy:

production-account (aws-account)
|__ us-east-1 (aws-region)
    |__ production-us-east-1-vpc (vpc)
        |__ production (environment)
            |__ ssh-bastion (service) 
               |__ remote.sh <~~~~~ YOU ARE HERE

Dynamic State Management

The directory structure is one of the cornerstones of this design as it enables us to dynamically generate the S3 key for a service’s state file. Within the remote.sh script (shown below) we parse the directory structure and then set/get our remote state.

A symlink of /templates/service-remote.sh to remote.sh is created in each service folder.

#!/bin/bash -e
# Must be run in the service's directory.

help_message() {
  echo -e "Usage: $0 [apply|destroy|plan|refresh|show]\n"
  echo -e "The following arguments are supported:"
  echo -e "\tapply   \t Refresh the Terraform remote state, perform a \"terraform get -update\", and issue a \"terraform apply\""
  echo -e "\tdestroy \t Refresh the Terraform remote state and destroy the Terraform stack"
  echo -e "\tplan    \t Refresh the Terraform remote state, perform a \"terraform get -update\", and issues a \"terraform plan\""
  echo -e "\trefresh \t Refresh the Terraform remote state"
  echo -e "\tshow    \t Refresh and show the Terraform remote state"
  exit 1
}

apply() {
  plan
  echo -e "\n\n***** Running \"terraform apply\" *****"
  terraform apply
}

destroy() {
  plan
  echo -e "\n\n***** Running \"terraform destroy\" *****"
  terraform destroy
}

plan() {
  refresh
  terraform get -update
  echo -e  "\n\n***** Running \"terraform plan\" *****"
  terraform plan
}

refresh() {

  account=$(pwd | awk -F "/" '{print $(NF-4)}')
  region=$(pwd | awk -F "/" '{print $(NF-3)}')
  vpc=$(pwd | awk -F "/" '{print $(NF-2)}')
  environment=$(pwd | awk -F "/" '{print $(NF-1)}')
  service=$(pwd | awk -F "/" '{print $NF}')

  echo -e "\n\n***** Refreshing State *****"

  terraform remote config -backend=s3 \
                          -backend-config="bucket=${account}" \
                          -backend-config="key=${region}/${vpc}/${environment}/${service}/terraform.tfstate" \
                          -backend-config="region=us-east-1"
}

show() {
  refresh
  echo -e "\n\n***** Running \"terraform show\"  *****"
  terraform show
}

## Begin script ##
if [ "$#" -ne 1 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
  help_message
fi

ACTION="$1"

case $ACTION in
  apply|destroy|plan|refresh|show)
    $ACTION
    ;;
  ****)
    echo "That is not a vaild choice."
    help_message
    ;;
esac

Service Module Instantiation

In addition to remote.sh, the environment’s service directory contains a main.tf file.

module "environment" {
  source = "../"
}

module "bastion" {
  source = "git@github.com:TerraformDesignPattern/bastionhost.git"

  aws_account      = "${module.environment.aws_account}"
  aws_region       = "${module.environment.aws_region}"
  environment_name = "${module.environment.environment_name}"
  hostname         = "${var.hostname}"
  image_id         = "${var.image_id}"
  vpc_name         = "${module.environment.vpc_name}"
}

We call two modules in main.tf. The first is an environment module, which we’ll talk about in a moment, and the second is an environment service module.

Environment Service Module

Environment service module?

Everytime I’ve introduced this term, I’ve seen this…

An environment service module, or ESM for short, is just a way to specify, in conversation, that we are talking about the code that actually instantiates a service module.

If you look at the ESM declaration in main.tf above, you’ll see it is using the output from the enviroment module to define variables that will be passed into the service module. If we take a step back to review our directory structure we see the service we are deploying sits within the production environment’s directory:

sysadvent-production/us-east-1/production-us-east-1-vpc/production/ssh-bastion

Within the production environment’s directory is an outputs.tf file.

output "aws_account" {
  value = "production"
}

output "aws_region" {
  value = "us-east-1"
}

output "environment_name" {
  value = "production"
}

output "vpc_name" {
  value = "production-us-east-1-vpc"
}

We are able to create an entire service, regardless of resources, with a very generic ESM and just four values from our environment module. We are using our organization’s defined and somewhat colloquial terms to create our infrastructure. We don’t need to remember ARNs, ID’s or other allusive information. We don’t need to remember naming conventions either as the service module will take care of this for us.

Service Module Example

So far we’ve established a repeatable ways to run our Terraform command and guarantee that our state is managed properly and consistently. We’ve also instantiated a service module from within an environment service module. We are now going to dive into the components of a service module.

A service module will be reused throughout your infrastrcuture so it must be generic and parameterized. The module will create all Terraform provider resources required by the service.

In my experience, I’ve found splitting each resource type into its own file improves readability. Below is the list of Terraform files from our example bastion host service module repository:

bastionhost
|-- data.tf
|-- ec2.tf
|-- LICENSE
|-- outputs.tf
|-- providers.tf
|-- route53.tf
|-- security_groups.tf
`-- variables.tf

The contents of most of these files will look pretty generic to the average Terraform user. The power of this pattern lies within the data.tf as it allows the simplistic instantiation.

// Account Remote State
data "terraform_remote_state" "account" {
  backend = "s3"

  config {
    bucket = "${var.aws_account}"
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

// VPC Remote State
data "terraform_remote_state" "vpc" {
  backend = "s3"

  config {
    bucket = "${var.aws_account}"
    key    = "${var.aws_region}/${var.vpc_name}/terraform.tfstate"
    region = "us-east-1"
  }
}

Sooooo. What populates the state file for the VPC data resources? Enter the VPC Service Module

There is no cattle.
There are no layers.
There is no spoon.

Everything is just a compartmentalized service. The module that creates your VPC is a separate “service” that lives in its own repository.

We create our account resources (DNS zones, SSL Certs, Users) within an account service module, our vpc resources from a VPC module within a vpc service module and our services (Application Services, RDS, Web Services) within an Environment Service Module.

We use a Bash wrapper to publish the state of resources in a consistent fashion.

Lastly, we abstract the complexity of infrastructure configuration management by querying Terraform state files based on a strict S3 key structure.

Real World Example

Follow along with the example by pulling the TerraformDesignPattern/environments repository. The configuration of each module within the environment repository will consist of roughly the same steps:

1. Create the required files. Usually main.tf and variables.tf or simply an outputs.tf file.
2. Populate variables.tf/outputs.tf with your desired values.
3. Create a symlink to a specific remote.sh (account-remote.sh, service-remote.sh or vpc-remote.sh) from within the appropriate directory.
4. For example, to create the remote.sh wrapper for your account service module, issue the following from within your environments/$ACCOUNT directory: ln -s ../templates/account-remote.sh remote.sh
5. Run ./remote.sh apply

Prerequisites

• Domain Name: I went over to Namecheap.com and grabbed the sysadvent.host domain name for $.88.
• State File S3 Bucket: Create the S3 bucket to store your state files in. This should be the name of the account folder within your environments repository. For this example I created the sysadvent-production S3 bucket.
• SSH Public Key: As of the writing of this post, the aws_key_pair Terraform resource does not currently support creating a public key, only importing one.
• SSL ARN: AWS Certificate Manager offers free SSL/TLS certificates.

Getting Started

This Real World Example assumes you are provisioning a new AWS account and domain. For those working in a brown field, the following section provides a quick example of how to build a scaffolding that can be used to deploy the design pattern.

State Scaffolding or Fake It ’Til You Make It (With Terraform)

Within the environments/sysadvent-production account directory is an s3.tf file that creates a dummy_object in S3:

resource "aws_s3_bucket_object" "object" {
  bucket = "${var.aws_account}"
  key    = "dummy_object"
  source = "outputs.tf"
  etag   = "${md5(file("outputs.tf"))}"
}

A new object will be uploaded when outputs.tf is changed. This change updates the remote state file and thus any outputs that have been added to outputs.tf will be added to the remote state file as well. To use a resource (IAM Role, VPC ID, or Zone ID) that was not created with Terraform, simply add the desired data to the account, vpc, or ESM’s outputs.tf file. Since not all resources can be imported via data resources, this enables us to migrate in small, iterable phases.

In the example below, the AWS Account ID will be added to the account’s state file via this mechanism. The outputs.tf file defines the account ID via the aws_account variable:

output "aws_account" {
  value = "${var.aws_account}"
}

Stage One: Create The Account Service Module (ASM)

Working Directory: environments/sysadvent-production

As per the name, this is were account wide resources are created such as DNS Zones or Cloudtrail Logs. The sysadvent-production ASM will create the following:

• Cloudwatch Log Stream for the account’s Cloudtrail
• Import a public key
• Route53 Zone
• The “scaffolding” S3 dummy_object from outputs.tf to publish:
• AWS Account ID
• Domain Name
• SSL ARN

Populate Variables

Populate the variables in the account’s variables.tf file:

aws_account - name of the account level folder
aws_account_id - your AWS provided account ID
domain_name - your choosen domain name
key_pair_name - what you want to name the key you are going to import
ssl_arn - ARN of the SSL certificate you created, free, with Amazon's Certificate Manager
public_key - the actual public key you want to import as your key pair

Execute!

Once you have created a state file S3 bucket and populated the variables.tf file with your desired values run ./remote.sh apply.

Stage Two: Create The VPC Service Module (VSM)

Working Directory: environments/sysadvent-production/us-east-1/production-us-east-1-vpc

The TerraformDesignPattern/vpc module creates the following resources:

• VPC
• Enable VPC Flow Logs
• Cloudwatch Log Stream for the VPC’s Flow Logs
• Flow Log IAM Policies
• An Internet Gateway
• Three private subnets
• Three public subnets with nat gateways and elastic IP addresses
• Routes, route tables, and associations

Populate Variables

Populate the following variables in the VSM’s variables.tf file:

availability_zones
aws_region
private_subnets
public_subnets
vpc_cidr
vpc_name

Execute!

Once you have populated the variables.tf file with your desired values, create the resources by running ./remote.sh apply.

Stage Three: Create The Environment Module

Working Directory: environments/sysadvent-production/us-east-1/production-us-east-1-vpc/production

The environment module stores the minimal amount of information required to pass to an environment service module. As mentioned previously, this module consists of a single outputs.tf file which requires you to configure the following:

aws_account
aws_region
environment_name
vpc_name

Stage Four: Create An Environment Service Module (ESM)

Working Directory: environments/sysadvent-production/us-east-1/production-us-east-1-vpc/production/elk

Congratulations, you’ve made it to the point where this wall of text really pays off.

Create the main.tf file. The ELK’s ESM calls an environment module, an ami_image_id module , and the ELK service module. The environment module supplies environment specific data such as the AWS account, region, environment name and VPC name. This module data is, in turn, passed to the ami_image_id module. The ami_image_id module will return AMI ID’s based on the enviroment’s region.

The ELK ESM will create the following resources:

• Three instance ELK stack within the previously created private subnet.
• The Elasticsearch instances will be clustered via the EC2 discovery plugin.
• A public facing ELB to access Kibana.
• A Route53 DNS entry pointing to the ELB.

Execute!

Once you created the main.tf file, create the resources by running ./remote.sh apply.

Appendix

Links

• Terraforn Design Pattern Github Organization
• Bastion Host Service Module
• Cloud Trail Module
• Environments Module
• ELK Service Module
• Packer Service Module
• VPC Module
• Terraform Style Guide

Gotchas

During the development of this pattern I stumbled across a couple gotchas. I wanted to share these with you but didn’t think they were necessarily pertinent to an introductory article.

Global Service Resources

Example: IAM Roles

We want to keep the creation of IAM roles in a compartmentalized module but IAM roles are global thus you can only create them once. Using count and a lookup based on our region, we can tell Terraform to only create the IAM role in a single region.

Example iam.tf file:

...
count = "${lookup(var.create_iam_role, var.aws_region, 0)}"
...

Example variables.tf file:

...
variable "create_iam_role" {
  default = {
    "us-east-1" = 1
  }
}
...

Referencing an ASM or VSM From an ESM

The ASM and VSM create account and VPC resources, respectively. If you were to reference an ASM or VSM a la an environment module within an ESM, you’ll essentially be attempting to recreate the resources originally created by the ASM and VSM.