Day 20 - Importing and manipulating your Terraform configuration

By: Paul Puschmann (@ppuschmann)
Edited by: Scott Murphy (@ovsage

What is terraform?

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.

Configuration files describe to Terraform the components needed to run a single application or your entire datacenter. Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.

The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.

The key features of Terraform are:

• Infrastructure as Code
• Execution Plans
• Resource Graph
• Change Automation

Source: Introduction to Terraform

On december 5th 2019 Kief Morris already pointed out on SysAdvent that you should
“break up your Terraform setup before it breaks you”
and references a Hashicorp-Video with Nicki Watt about the proposed way a e-commerce system should evolve: Evolving Your Infrastructure with Terraform

I like the approach and the presentation and would like to share some of our experiences:

We already had an elaborate Terraform setup that was complex and contained some duplicate code and definitions having separate folders for our different staging environments.
This setup got converted to a terragrunt configuration which resulted technically in very “DRY” code.
But by this we also had some additional burdens to take:

We had to take care of the changes of another tool (terragrunt) and had to deal with very complex code:
Our code had modules nested into modules containing another layer of modules. The vast number of variable redirections created by this nesting of modules was slowing us down and prevented the further effective development of new features. For example the onboarding of new colleagues to this
very dry pile of code was consuming much time and other colleagues were reluctant to do deeper changes at all.

With the changes in the upcomming Terraform 0.12 we decided to take a step back and give the code a fresh start while keeping the infrastructure itself up and running. Our goals were:

• to have understandable code (with the compromise of having duplicate code)
• to be able to do changes to configuration
• to actually own the code as a team and share the responsibility

After making up our minds we agreed on the following strategy:

1. Create plain, unoptimized terraform-configration that ressembled the current state of our project
2. Split the configuration into separated configuration domains
3. Introduce terraform-modules to reduce the amount of duplicated code and definitions

The following examples will show you one way to get these tasks done. The third step, creation of terraform-modules, is left out to reduce the size of this article.

Importing

We discussed the following approaches for the configuration rewrite:

• Doing completely manual imports with terraform import.
• Use terraformer to generate config-files and create new statefiles.
• Try to apply an empty configuration and parse the verbose diff to actually create new configuration files.

We chose the third approach (because we can), which is just a more automatic version of the first approach.

The automated way: terraformer

Terraformer is a CLI tool that generates .tf and .tfstate files based on existing infrastructure (reverse Terraform).

Terraformer may not support all the components you use, but will perhaps cover a great deal of them.

Example

Inside on an empty directory create a .tf-file with this input:

provider "google" {
}

Execute terraform init to download and initialize the required Terraform providers:

Execute terraformer with parameters to import your current live-configuration:

terraformer import google --regions=europe-west1 --projects=myexample-project-1 --resources=addresses,instances,disks,firewalls
2019/12/07 22:13:11 google importing project myexample-project-1 region europe-west1
2019/12/07 22:13:13 google importing... addresses
2019/12/07 22:13:14 Refreshing state... google_compute_address.tfer--ext-myexample-webserver
2019/12/07 22:13:16 google importing... instances
2019/12/07 22:13:17 Refreshing state... google_compute_instance.tfer--myexample-webserver
2019/12/07 22:13:19 google importing... disks
2019/12/07 22:13:21 Refreshing state... google_compute_disk.tfer--europe-west1-b--myexample-webserver-data
2019/12/07 22:13:21 Refreshing state... google_compute_disk.tfer--europe-west1-b--myexample-webserver
2019/12/07 22:13:22 google importing... firewalls
2019/12/07 22:13:23 Refreshing state... google_compute_firewall.tfer--default-allow-ssh
2019/12/07 22:13:23 Refreshing state... google_compute_firewall.tfer--fw-i-myexample-webserver-ssh
2019/12/07 22:13:23 Refreshing state... google_compute_firewall.tfer--fw-i-myexample-webserver-web
2019/12/07 22:13:23 Refreshing state... google_compute_firewall.tfer--default-allow-icmp
2019/12/07 22:13:23 Refreshing state... google_compute_firewall.tfer--default-allow-internal
2019/12/07 22:13:25 google Connecting....
2019/12/07 22:13:25 google save addresses
2019/12/07 22:13:25 google save tfstate for addresses
2019/12/07 22:13:25 google save instances
2019/12/07 22:13:25 google save tfstate for instances
2019/12/07 22:13:25 google save disks
2019/12/07 22:13:25 google save tfstate for disks
2019/12/07 22:13:25 google save firewalls
2019/12/07 22:13:25 google save tfstate for firewalls

The resulting tree in the filesystem looks like this:

.
├── generated
│   └── google
│       └── myexample-project-1
│           ├── addresses
│           │   └── europe-west1
│           │       ├── compute_address.tf
│           │       ├── outputs.tf
│           │       ├── provider.tf
│           │       └── terraform.tfstate
│           ├── disks
│           │   └── europe-west1
│           │       ├── compute_disk.tf
│           │       ├── outputs.tf
│           │       ├── provider.tf
│           │       └── terraform.tfstate
│           ├── firewalls
│           │   └── europe-west1
│           │       ├── compute_firewall.tf
│           │       ├── outputs.tf
│           │       ├── provider.tf
│           │       └── terraform.tfstate
│           └── instances
│               └── europe-west1
│                   ├── compute_instance.tf
│                   ├── outputs.tf
│                   ├── provider.tf
│                   └── terraform.tfstate
└── terraform.tf

Taking a look into the results:

Head of file generated/google/myexample-project-1/firewalls/europe-west1/compute_firewall.tf

resource "google_compute_firewall" "tfer--default-allow-icmp" {
  allow {
    protocol = "icmp"
  }

  description    = "Allow ICMP from anywhere"
  direction      = "INGRESS"
  disabled       = "false"
  enable_logging = "false"
  name           = "default-allow-icmp"
  network        = "https://www.googleapis.com/compute/v1/projects/myexample-project-1/global/networks/default"
  priority       = "65534"
  project        = "myexample-project-1"
  source_ranges  = ["0.0.0.0/0"]
}

Head of file generated/google/myexample-project-1/instances/europe-west1/compute_instance.tf

resource "google_compute_instance" "tfer--myexample-webserver" {
  attached_disk {
    device_name = "myexample-webserver-data"
    mode        = "READ_WRITE"
    source      = "https://www.googleapis.com/compute/v1/projects/myexample-project-1/zones/europe-west1-b/disks/myexample-webserver-data"
  }

  boot_disk {
    auto_delete = "true"
    device_name = "persistent-disk-0"

    initialize_params {
      image = "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-9-stretch-v20190326"
      size  = "20"
      type  = "pd-standard"
    }

    mode   = "READ_WRITE"
    source = "https://www.googleapis.com/compute/v1/projects/myexample-project-1/zones/europe-west1-b/disks/myexample-webserver"
  }

  can_ip_forward      = "false"
  deletion_protection = "false"
  enable_display      = "false"

  labels = {
    ansible-group = "webserver"
  }

  machine_type = "n1-standard-2"

  metadata = {
    enable-oslogin = "TRUE"
  }

  name = "myexample-webserver"

  network_interface {
    access_config {
      nat_ip       = "315.256.10.276"
      network_tier = "PREMIUM"
    }

    name               = "nic0"
    network            = "https://www.googleapis.com/compute/v1/projects/myexample-project-1/global/networks/default"
    network_ip         = "10.132.0.10"
    subnetwork         = "https://www.googleapis.com/compute/v1/projects/myexample-project-1/regions/europe-west1/subnetworks/default"
    subnetwork_project = "myexample-project-1"
  }

  project = "myexample-project-1"

  scheduling {
    automatic_restart   = "true"
    on_host_maintenance = "MIGRATE"
    preemptible         = "false"
  }

  service_account {
    email  = "998619246879-compute@developer.gserviceaccount.com"
    scopes = ["https://www.googleapis.com/auth/compute.readonly"]
  }

  tags = ["webserver", "test", "sysadvent"]
  zone = "europe-west1-b"
}

The results are impressive for the supported services.

After this you might spend some time rewriting the configuration and moving resources between statefiles.
At least now you have some valid configurations you can edit and do new deployments upon and also valid statefiles.

terraformer will also generate configurations for resources with lifecycle-attributes like deletion-prevention, a bonus compared to the manual imports.

The hard way: manual imports

A different approach could be to start with a nearly empty Terraform-file:

provider "google" {
}

Excuting terraform plan you’d now get a list of resources to get removed.
With some grep and sed magic you can recreate your resource definition.

But note: Resources that are using a lifecycle attribute to prevent the deletion of this item will not get mentioned in the diff created by terraform plan.

Example:

resource "google_compute_address" "ext_address_one" {
  name    = "ext-address-one"

  lifecycle {
    prevent_destroy = true
  }
}

Conclusion on imports

Imports can save time on recovery or when transforming configuration.

Independent of the method you use to import or recreate your configuration, limitations will apply.
The generated code or diff will not honour Terraform modules that were possibly used to create the resources,
but will create static resource-definitions.
Values will get hardcoded into the resource-definition, for example with external IP-addresses.

In short: You won’t get perfect Terraform configuration with an import, but you at least you’ll be some steps ahead.

Working with statefiles

Especially when using terraformer to import your configuration and generate code, you’ll find yourself with a set of configuration and statefiles,
one of each per resource type and region.
This generated code is functional, but far away from a structure you want to work with.

The terraform state command provides a versatile set of subcommands to manipulate Terraform statefiles.

With the help of terraform state mv you can rename
resources or move resources to different statefiles.
This command also allows you to move resources in and out of modules.

Other useful commands are terraform state pull and terraform state push to pull or push the statefile
from the configured storage backend.

Moving away from “terralith”

I’d like to show some methods on how to move between some of the different models Nicki Watt describes.

The name terralith is a synonym for a big Terraform-configuration that contains items of various infrastructure domains,
possibly a complete project.
Ideally you’d like to change your Terraform configuration without breaking your application or environments.

For demonstration purposes I created a setup of two MySQL-instances and six webservers, using modules.
Configuration of these instance-types is bundled in a terralith, meaning there’s one statefile for the complete
setup of the project: firewall-rules, GCE-instances, NAT-gateway, DNS-setup and other configuration.

Of course using modules is one way to move away from terralith, but I’d like to show you a different way to go first:
Moving on by splitting configuration into multiple, domain-separated blocks without increasing the technical complexity at the same time.

The first cut of configuration in the following example will be the separation of the “server” related parts from the general parts of the configuration.
The current setup will stay in a directory main, the new separeted setup will be located in the directory servers right next to main.

The current resources created in the statefile of our main project:

$ terraform state list
data.google_compute_zones.available
data.google_project.project
data.terraform_remote_state.mydemo
google_compute_address.bastionhost
google_compute_address.nat_gateway
google_compute_firewall.ext_to_bastionhost
google_compute_firewall.intern_to_ext
google_compute_firewall.bastion_to_intern
google_compute_router.nat_gateway
google_compute_router_nat.nat_gateway
module.bastionhosts.data.google_compute_zones.available
module.bastionhosts.google_compute_disk.instances[0]
module.bastionhosts.google_compute_instance.instances[0]
module.bastionhosts.google_dns_record_set.instances[0]
module.bastionhosts.google_dns_record_set.instances-private[0]
module.mysql.data.google_compute_zones.available
module.mysql.google_compute_disk.additional_v2["mysql-1-varlibmysql"]
module.mysql.google_compute_disk.additional_v2["mysql-2-varlibmysql"]
module.mysql.google_compute_disk.instances_v2["mysql-1"]
module.mysql.google_compute_disk.instances_v2["mysql-2"]
module.mysql.google_compute_instance.instances_v2["mysql-1"]
module.mysql.google_compute_instance.instances_v2["mysql-2"]
module.mysql.google_dns_record_set.instances-private_v2["mysql-1"]
module.mysql.google_dns_record_set.instances-private_v2["mysql-2"]
module.webservers.data.google_compute_zones.available
module.webservers.google_compute_disk.instances_v2["web-1"]
module.webservers.google_compute_disk.instances_v2["web-2"]
module.webservers.google_compute_disk.instances_v2["web-3"]
module.webservers.google_compute_disk.instances_v2["web-4"]
module.webservers.google_compute_disk.instances_v2["web-5"]
module.webservers.google_compute_disk.instances_v2["web-6"]
module.webservers.google_compute_instance.instances_v2["web-1"]
module.webservers.google_compute_instance.instances_v2["web-2"]
module.webservers.google_compute_instance.instances_v2["web-3"]
module.webservers.google_compute_instance.instances_v2["web-4"]
module.webservers.google_compute_instance.instances_v2["web-5"]
module.webservers.google_compute_instance.instances_v2["web-6"]
module.webservers.google_dns_record_set.instances-private_v2["web-1"]
module.webservers.google_dns_record_set.instances-private_v2["web-2"]
module.webservers.google_dns_record_set.instances-private_v2["web-3"]
module.webservers.google_dns_record_set.instances-private_v2["web-4"]
module.webservers.google_dns_record_set.instances-private_v2["web-5"]
module.webservers.google_dns_record_set.instances-private_v2["web-6"]

The state is saved in a Google storage bucket.

terraform {
  backend "gcs" {
    bucket = "mydemoproject"
    prefix = "dev/main"
  }
}

Example of the configured module webservers:

module "webservers" {
  source = "git::ssh://github.com/<sorry-this-only-an-example>.git"

  instance_map = {
    "web-1" : { zone = "europe-west1-b" },
    "web-2" : { zone = "europe-west1-c" },
    "web-3" : { zone = "europe-west1-d" },
    "web-4" : { zone = "europe-west1-b" },
    "web-5" : { zone = "europe-west1-c" },
    "web-6" : { zone = "europe-west1-d" },
  }

  region              = var.region
  machine_type        = "n1-standard-4"
  disk_size           = "15"
  disk_image          = var.image
  subnetwork          = var.subnetwork
  subnetwork_project  = var.network_project
  tags                = ["webserver-dev"]
  label_ansible_group = "webserver"

  dns_domain_intern       = var.internal_dnsdomain
  dns_managed_zone_intern = var.internal_managedzone

  project         = var.project
  network_project = var.network_project
}

We first create a new directory for the new MySQL-instance and webserver configuration, called servers, add the required
files for variables and providers, move over the module-configuration, and point the state to a different file:

terraform {
  backend "gcs" {
    bucket = "mydemoproject"
    prefix = "dev/servers"
  }
}

After the initialization of this Terraform configuration with terraform init you can execute terraform plan
to check what Terraform currently thinks is needed to change. The plan is to create two MySQL-instances
and six webservers, because the statefile of this configuration is still empty or doesn’t have the
actual information of the running instances.

Execute the following commands in the main directory to create a local copy of the remote statefile and
then move the states of the named modules from the local statefile to the new file ../servers/default.tfstate:

$ terraform state pull | tee default.tfstate
$ terraform state mv -state=default.tfstate -state-out=../servers/default.tfstate 'module.webservers' 'module.webservers'
Move "module.webservers" to "module.webservers"
Successfully moved 1 object(s).
$ terraform state mv -state=main.tfstate -state-out=../servers/default.tfstate 'module.mysql' 'module.mysql'
Move "module.mysql" to "module.mysql"
Successfully moved 1 object(s).

After successful creation of a new statefile in your new configuration directory do this:

Upload the statefile with terraform state push default.tfstate to your configured storage backend.
Then move the configuration for the resources you just moved to the new state to the servers directory as well.

Execute terraform plan to get this output:

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Move back to the directory main and push the changed statefile to remote: terraform state push default.tfstate

Delete your local statefiles for cleanup and you’re done.

Conclusion

Thank you for reading all this. I hope I helped you to understand the first steps away from a terralith to a more modular setup.
Unfortunately, a more detailed explanation of working with modules, as well as listing the pros and cons is beyond the scope of this article.

As you can see, working with Terraform can be more than just terraform init, terraform plan & terraform apply. Changes on the scope of the configuration do not automatically mean to destroy and recreate everything.