By: Philip Brocoum (@stedwick)
Edited by: Ryan Hass (@_ryanhass_)
Here at Syncta we have a classic monolithic Ruby on Rails application with roughly 150 models. For the past six months, I have been working on reimagining our deployment on Kubernetes.
I began this project with two principles in mind:
1. Picking the best tool for each job.
2. Each tool should follow the Linux philosophy of doing one thing, and doing it well.
The first thing I asked myself was, "Can we containerize our Rails app?" The most difficult thing was the learning curve related to Docker, but the answer was, “Yes!” Our Gemfile is 200 lines long, and our Docker image weighs in around 1 GB, but I've seen bigger. Our Dockerfile starts with `FROM bitnami/ruby` and includes `nodejs libpq-dev file graphviz busybox vim` for compiling native gem extensions. I chose the Bitnami image because I tried Alpine first and found myself installing too many prerequisites. We push these images to Quay.io, which is a very user-friendly image repository. For example, they offer copy-paste credentials for installing the imagePullSecret into k8s.
Furthermore, I created a docker-compose.test.yml that runs all ~1000 of our tests completely self-contained; PostgreSQL, Redis, ElasticSearch, and ChromeDriver all run inside their respective containers for our test suite. The biggest change was that we used to simply install Chrome on our virtual machine, and use it locally for our tests. We now have to run our browser in a container using Browserless.io.
The next step was to get this Docker Composition running in a CI/CD pipeline, for which we used Bitbucket Pipelines. Running everything within Docker (with its overhead) on one CI instance used up the standard 4 GB in Bitbucket Pipelines. As a result we had to upgrade to the 2x memory build in order to run Chrome. Our test suite now automatically runs in about 15 minutes on a push to Bitbucket. Here's a screenshot of what our pipeline looked like previously in Semaphore, and now in Pipelines.
Now the real fun begins: we have a working, tested, Dockerized version of our monolithic Rails app, and the next step is to deploy it to Kubernetes. I like to keep things as simple as possible. With that in mind, we used Spotinst to spin up a k8s cluster on AWS EKS. Spotinst handles everything, including installation and cluster auto-scaling, and then we spun up a Rancher instance for an easy-to-use GUI and attached it to our k8s cluster using the provided Spotinst credentials. Rancher allows us to one-click-install things like Traefik & Grafana (see screenshot).
Finally, we launched a deployment inside the k8s cluster using the Docker image we created in step one. Wow! Right now, only our secondary staging server is running on k8s, from a git branch parallel to master and Rancher is currently a single node install. There's still plenty of QA to do. We are planning a Canary rollout next year.
There are some good lessons to learn from this whole journey. First, it's far easier to develop/test/deploy when everything is Dockerized and guaranteed to be identical. However, the downside is that there is a huge learning curve with Docker and k8s that our devs will need to get up to speed with. In order to help mitigate the learning curve I have started a Makefile with many common commands that our developers use. My New Year's wish for SysAdvent is for a smooth rollout to production in 2020. :-)
help:
@echo Syncta Main Rails Makefile
@echo ...
dev:
@kubectl run -n "smr-development" \
--image="${KI}" \
...
Specs:
@export COMPOSE_PROJECT_NAME="smrtest"; \
config/devops/specs.sh
If any of this sounds interesting to you, and you'd like to join our team, please get in touch with our recruiter, Jimmy! We are a remote-first development team (headquartered in Portland, OR). We are looking for a Senior/Lead Full-Stack Developer with good experience in Ruby on Rails immediately. In 2020, we'll be hiring another Developer and a Project Manager. Syncta is a small startup working on backflow and other water-related technologies, and we were recently acquired by Watts Water Technologies. I'm Philip Brocoum, Head of Development at Syncta, and Happy SysAdvent to all!
No comments:
Post a Comment