Day 13 - Ephemeral PR Environments: Enabling automated testing at a rapid pace

By: Amar Sattaur
Edited by: Jennifer Davis (@sigje)

Recently, I've been thinking a lot about how to implement the concepts of least privilege while also speeding up the feedback cycle in the developer workflow. However, these two things are not very quickly intertwined. Therefore, there needs to be underlying tooling and visibility to show developers the data they need for a successful PR merge.

A developer doesn't care about what those underlying tools are; they just want access to a system where they can:

• See the logs of the app that they're making a change for and the other relevant apps
• See the metrics of their app so they can adequately gauge performance impact

One way to achieve this is with ephemeral environments based on PR's. The idea is that the developer opens up a PR and then automatically a new environment is spun up based on provided defaults with the conditions that the environment is:

• deployed in the same way that dev/stage/prod are deployed, just with a few key elements different
• labeled correctly so that the NOC/Ops teams know the purpose of these resources
• Integrated with logging/metrics and useful tags so that the engineer can easily see metrics for this given PR build

That sounds like a daunting task but through the use of Kubernetes, Helm, a CI Platform (GitHub Actions in this tutorial) and ArgoCD, you can make this a reality. Let's look at an example application leveraging all of this technology.

Example app

You can find all the code readily available in this GitHub Repo.

Pre-requisites Used in this Example

Tool	Version
kubectl	v1.21
Kubernetes Cluster	v1.20.9
Helm	v3.6.3
ArgoCD	v2.0.5
kube-prometheus-stack	v0.50.0

The example app that you’re going to deploy today is a Prometheus exporter that exports a custom metric with an overridable label set:

• The `version` of the deployed app
• The `branch` of the PR
• The PR ID

Pipeline

Now that I've defined the goal, let's go a little more in-depth on how you'll get there. First, let's take a look at the PR pipeline in .github/workflows/pull_requests.yml:

---
name: 'Build image and push PR image to ghcr'
on:
  pull_request:
    types: [assigned, opened, synchronize, reopened]
    branches:
      - main

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Build image
        uses: docker/build-push-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}
          tags: PR-${{ github.event.pull_request.number }}
       

This pipeline runs on pull requests events to the main branch. So, when you open a PR, push a commit to an existing PR, reopen a closed PR, or assign it to a user, this pipeline will get triggered. It defines two workflows, the first of which is build. It's relatively straightforward: take the Dockerfile that lives in the root of your repo and build a container image out of it and tag it for use with GitHub Container Registry. The tag is the PR ID of the triggering pull request.

The second workflow is the one where we deploy to ArgoCD:

 deploy:
    needs: build
    container: ghcr.io/jodybro/argocd-cli:1.1.0
    runs-on: ubuntu-latest
    steps:
      - name: Log into argocd
        run: |
          argocd login ${{ secrets.ARGOCD_GRPC_SERVER }} --username ${{ secrets.ARGOCD_USER }} --password ${{ secrets.ARGOCD_PASSWORD }}
      - name: Deploy PR Build
        run: |
          argocd app create sysadvent2021-pr-${{ github.event.pull_request.number }} \
            --repo https://github.com/jodybro/sysadvent2021.git \
            --revision ${{ github.head_ref }} \
            --path . \
            --upsert \
            --dest-namespace argocd \
            --dest-server https://kubernetes.default.svc \
            --sync-policy automated \
            --values values.yaml \
            --helm-set version="PR-${{ github.event.pull_request.number }}" \
            --helm-set name="sysadvent2021-pr-${{ github.event.pull_request.number }}" \
            --helm-set env[0].value="PR-${{ github.event.pull_request.number }}" \
            --helm-set env[1].value="${{ github.head_ref }}" \
            --helm-set env[2].value="sysadvent2021-pr-${{ github.event.pull_request.number }}"
       

This workflow runs a custom image that I wrote that wraps the argocd cli tool in a container and allows for arbitrary commands to be executed against an authenticated ArgoCD instance.

It then creates a Kubernetes object of kind: Application which is a CRD that ArgoCD deploys into your cluster to define where you want to pull the application from and how to deploy it (helm/kustomize etc..).

Putting it all together

Now, let's see this pipeline in action. First, head to your repo and create a PR against the main branch with some changes; it doesn't matter what the changes are as all PR events will trigger the pipeline.

You can see that my PR has triggered a pipeline which can be viewed here. Furthermore, you can see that this pipeline was executed successfully, so if I go to my ArgoCD instance, I would see an application with this PR ID.

So, if you are following along, now you have two deployments of this example app, one should show labels for the main branch, and one should show labels for the PR branch.

Let's verify by port-forwarding to each and see what you get back.

Main branch

First, let's check out the main branch application:

kubectl port-forward service/sysadvent2021-main 8000:8000 
Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000
       

As you can see, the branch is set to main with the correct version.

And if you check out the state of our Application in ArgoCD:

Everything is healthy!

PR

Now let's check the PR deployment:

kubectl port-forward service/sysadvent2021-pr-1 8000:8000 
Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000
       

This one's labels are showing the branch and the version from the PR.

This pod returns:

And in ArgoCD:

Final thoughts

It really is that easy to get PR environments running in your company!

Resources

* Source Code Repo