Day 16 - Merry PaaSmas and a Very Continuous Integration

Written by: Paul Czarkowski (@pczarkowski)
Edited by: Dan Phrawzty (@phrawzty)

Docker and the ecosystem around it have done some great things for developers, but from an operational standpoint, it's mostly just the same old issues with a fresh coat of paint. Real change happens when we change our perspective from Infrastructure (as a Service) to Platform (as a Service), and when the ultimate deployment artifact is a running application instead of a virtual machine.

Even Kubernates still feels a lot like IaaS - just with containers instead of virtual machines. To be fair, there are already some platforms out there that shift the user experience towards the application (Cloud Foundry and Heroku come to mind), but many of them have a large operations burden, or are provided in a SaaS model only.

In the Docker ecosystem we are starting to see more of these types of platforms, the first of which was Dokku which started as a single machine Heroku replacement written in about 100 lines of Bash. Building on top of that work other, richer systems like Deis and Flynn have emerged, as well as custom solutions built in-house, like Yelp's PaaSta.

Actions speak louder than words, so I decided to document (and demonstrate) a platform built from the ground up (using Open Source projects) and then deploy an application to it via a Continuous Integration/Deployment (CI/CD) pipeline.

You could (and probably would) use a public cloud provider for some (or all) of this stack; however, I wanted to demonstrate that a system like this can be built and run internally, as not everybody is able to use the public cloud.

As I wrote this I discovered that while figuring out the right combination of tools to run was a fun process, the really interesting stuff was building the actual CI/CD pipeline to deploy and run the application itself. This means that while I'll briefly describe the underlying infrastructure, I will not be providing a detailed installation guide.

Infrastructure

While an IaaS is not strictly necessary here (I could run Deis straight on bare metal), it makes sense to use something like OpenStack as it provides the ability to request services via API and use tooling like Terraform. I installed OpenStack across across a set of physical machines using Blue Box's Ursula.

Next the PaaS itself. I have familiarity with Deis already and I really like its (Heroku-esque) user experience. I deployed a three node Deis cluster on OpenStack using the Terraform instructions here.

I also deployed an additional three CoreOS nodes using Terraform on which I ran Jenkins using the standard Jenkins Docker image.

Finally, there is a three-node Percona database cluster running on the CoreOS nodes, itself fronted by a load balancer, both of which use etcd for auto-discovery. Docker images are available for both the cluster and the load balancer.

Ghost

The application I chose to demo is the Ghost blogging platform. I chose it because it's a fairly simple app with well-known backing service (MySQL). The source, including my Dockerfile and customizations, can be found in the paulczar/ci-demo GitHub repository.

The hostname and database credentials of the MySQL load balancer are passed into Ghost via environment variables (injected by Deis) to provide a suitable database backing service.

For development, I wanted to follow the GitHub Flow methodology as much as possible. My merge/deploy steps are a bit different, but the basic flow is the same. This allows me to use GitHub's notification system to trigger Jenkins jobs when Pull Requests are created or merged.

I used the Deis CLI to create two applications: ghost from the code in the master branch, and stage-ghost from the code in the development branch. These are my production and staging environments, respectively.

Both the development and master branches are protected with GitHub settings that restrict changes from being pushed directly to the branch. Furthermore, any Pull Requests need to pass tests before they can be merged.

Deis

Deploying applications with Deis is quite easy and very similar to deploying applications to Heroku. As long as your git repo has a Dockerfile (or supports being discovered by the cedar tooling), Deis will figure out what needs to be done to run your application.

Deploying an application with Deis is incredibly simple:

1. First you use deis create to create an application (on success the Deis CLI will add a remote git endpoint).
2. Then you run git push deis master which pushes your code and triggers Deis to build and deploy your application.

$ git clone https://github.com/deis/example-go.git
$ cd example-go
$ deis login http://deis.xxxxx.com
$ deis create helloworld 
Creating Application... ...
done, created helloworld
Git remote deis added
$ git push deis master

Counting objects: 39, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (38/38), done.
Writing objects: 100% (39/39), 5.17 KiB | 0 bytes/s, done.
Total 39 (delta 12), reused 0 (delta 0)

-----> Building Docker image
remote: Sending build context to Docker daemon 5.632 kB
<<<<<<<   SNIP   >>>>>>>
-----> Launching... 
       done, helloworld:v2 deployed to Deis
       http://helloworld.ci-demo.paulcz.net

Jenkins

After running the Jenkins Docker container I had to do a few things to prepare it:

1. Run docker exec -ti jenkins bash to enter the container and install the Deis CLI tool and run deis login which saves a session file so that I don't have to login on every job.
2. Add the GitHub Pull Request Builder (GHPRB) plugin.
3. Secure it with a password.
4. Run docker commit to commit the state of the Jenkins container.

I also had to create the jobs to perform the actual work. The GHPRB plugin made this fairly simple and most of the actual jobs were variations of the same script:

#!/bin/bash

APP="ghost"
git checkout master

git remote add deis ssh://git@deis.ci-demo.paulcz.net:2222/${APP}.git
git push deis master | tee deis_deploy.txt

Continuous Integration / Deployment

Local Development

Docker's docker-compose is a great tool for quickly building development environments (combined with Docker Machine it can deploy locally, or to the cloud of your choice). I have placed a docker-compose.yml file in the git repo to launch a mysql container for the database, and a ghost container:

ghost:
  build: .
  ports:
    - 5000:5000
  volumes:
    - .:/ghost
  environment:
    URL: http://localhost:5000
    DB_USER: root
    DB_PASS: ghost
  links:
    - mysql
mysql:
  image: percona
  ports:
   - "3306:3306"
  environment:
    MYSQL_ROOT_PASSWORD: ghost
    MYSQL_DATABASE: ghost

I also included an aliases file with some useful aliases for common tasks:

alias dc="docker-compose"
alias npm="docker-compose run --rm --no-deps ghost npm install --production"
alias up="docker-compose up -d mysql && sleep 5 && docker-compose up -d --force-recreate ghost"
alias test="docker run -ti --entrypoint='sh' --rm test /app/test"
alias build="docker-compose build"

Running the development environment locally is as simple as cloning the repo and calling a few commands from the aliases file. The following examples show how I added s3 support for storing images:

$ git clone https://github.com/paulczar/ci-demo.git
$ cd ci-demo
$ . ./aliases
$ npm
> sqlite3@3.0.8 install /ghost/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build
...
...
$ docker-compose run --rm --no-deps ghost npm install --save ghost-s3-storage
ghost-s3-storage@0.2.2 node_modules/ghost-s3-storage
├── when@3.7.4
└── aws-sdk@2.2.17 (xmlbuilder@0.4.2, xml2js@0.2.8, sax@0.5.3)
$ up

Docker Compose v1.5 allows variable substitution so I can pull AWS credentials from environment variables which means they don't need to be saved to git and each dev can use their own bucket etc. This is done by simply adding these lines to the docker-compose.yml file in the environment section:

ghost:
  environment:
    S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
    S3_ACCESS_KEY: ${S3_ACCESS_KEY}

I then added the appropriate environment variables to my shell and ran up to spin up a local development environment of the application. Once it was running I was able to confirm that the plugin was working by uploading the following image to the s3 bucket via the Ghost image upload mechanism:

Pull Request

All new work is done in feature branches. Pull Requests are made to the development branch of the git repo which Jenkins watches using the github pull request plugin (GHPR). The development process looks a little something like this:

$ git checkout -b s3_for_images
Switched to a new branch 's3_for_images'

Here I added the s3 module and edited the appropriate sections of the Ghost code. Following the GitHub flow I then created a Pull Request for this new feature.

$ git add .
$ git commit -m 'use s3 to store images'
[s3_for_images 55e1b3d] use s3 to store images
 8 files changed, 170 insertions(+), 2 deletions(-)
 create mode 100644 content/storage/ghost-s3/index.js
$ git push origin s3_for_images 
Counting objects: 14, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (14/14), 46.57 KiB | 0 bytes/s, done.
Total 14 (delta 5), reused 0 (delta 0)
To git@github.com:paulczar/ci-demo.git
 * [new branch]      s3_for_images -> s3_for_images

Jenkins will be notified when a developer opens a new Pull Request against the development branch and will kick off tests. Jenkins will then create and deploy an ephemeral application in Deis named for the Pull Request ID (PR-11-ghost).

The ephemeral environment can be viewed at http://pr-xx-ghost.ci-demo.paulczar.net by anyone wishing to review the Pull Request. Subsequent updates to the PR will update the deployed application.

We can run some manual tests specific to the feature being developed (such as uploading photos) once the URL to the ephemeral application is live.

Staging

Jenkins will see that a Pull Request is merged into the development branch and will perform two jobs:

1. Delete the ephemeral environment for Pull Request as it is no longer needed.
2. Create and deploy a new release of the contents of the development branch to the staging environment in Deis (http://stage-ghost.ci-demo.paulczar.net).

Originally when I started building this demo I had assumed that being able to perform actions on PR merges/closes would be simple, but I quickly discovered that none of the CI tools, that I could find, supported performing actions on PR close. Thankfully I was able to find a useful blog post that described how to set up a custom job with a webhook that could process the GitHub payload.

Production

Promoting the build from staging to production is a two step process:

1. The user who wishes to promote it creates a pull request from the development branch to the master branch. Jenkins will see this and kick off some final tests.

1. Another user then has to merge that pull request which will fire off a Jenkins job to push the code to Deis which cuts a new release and deploys it to the production environment (http://ghost.ci-demo.paulczar.net).

Conclusion

Coming from an operations background, I thought that figuring out how to build and run a PaaS from the metal up would be a really interesting learning exercise. It was! What I didn't expect to discover, however, was that actually running an application on that PaaS would be so compelling. Figuring out the development workflow and CI/CD pipeline was an eye-opener as well.

That said, the most interesting outcome of this exercise was increased empathy: the process of building and using this platform placed me directly in the shoes of the very developers I support. It further demonstrated that by changing the focus of the user experience to that person's core competency (the operator running the platform, and the developer using the platform) we allow the developer to "own" their application in production without them needing to worry about VMs, firewall rules, config management code, etc.

I also (re-)learned that while many of us default to cloud services such as AWS, Heroku, and Travis CI, there are solid alternatives that can be run in-house. I was also somewhat surprised at how powerful (and simple) Jenkins can be (even if it is still painful to automate).

I am grateful that Sysadvent gave me a reason to perform this little experiment. I learned a lot, and I hope that this article passes on some of that knowledge and experience to others.

Day 15 - Fear and Loathing in Systems Administration

Written by H. “Waldo” Grunenwald (@gwaldo)
Edited by Shaun Mouton (@sdmouton)

“DevOps Doesn’t Work”

The number of times that I’ve heard this is amazing. The best thing about this phrase is that the people who say it are often completely right, even if for very wrong reasons.

Who Says This?

Well, let’s talk about the people who most commonly have this reaction: SysAdmins. I’m going to use the term “SysAdmins” as a shorthand for a broad group. The people in this group have widely varying titles, but it is most commonly “Systems”, “Network”, or “Operations” follwed by “Administrator”, “Engineer”, “Technician”, or “Analyst”.

In some companies, these folks have the best computers in the place. In others, they have to live with the worst. Their workspace probably isn’t very nice, and almost certainly has no natural light. If there is a pager rotation, they are almost certainly on it. If there isn’t a rotation, they’re basically on-call all of the time.

During the course of a normal day they might have to switch contexts between disaster planning, calculating power and HVAC needs for a new datacenter, scrambling to complete an outage-driven SAN migration, rushing to address urgent requests to help people with their email, to troubleshoot a printing problem, or suss out why someone can’t get to their electric company’s bill pay website. They may be the sole person with database expertise in the company, or they may work on a team of dozens.

The work is largely invisible except when something fails, in which case it’s highly visible and widely impacting.

These are typically cynical people, because there are only so many times that you can’t make the team/department/company party for ostensibly “celebrating our successes” because something’s broken, and you’re left to clean up after the “success”. There are only so many times that one sees a new project announced and begins to hire more people. When asked who’s going to support the new project, the response is a blank look and “you are”. The “…of course.” may not be vocalized, but it’s probably there. When asked how many people they get to hire to help with the workload, the response is a combination of “sorry, but there wasn’t anything left in the budget”, “it won’t be that much more work”, or a variation of the “team player / good soldier” speech. There are only so many times one can take getting your requests for training or conference budget rejected out of hand, and have your requests for training or conference budget laughed out of the room.

They probably have basic working knowledge of a half-dozen programming languages, but most likely they often think in Shell. They probably know at least three ways of testing if a port is open, and probably have a soft spot in their heart for a couple of shell commands.

They may have seen or participated in a DevOps initiative that consisted of a team or position rename, or helpfully suggested that they install some Config Management and Monitoring software so that “we can DevOps now…” or “so we can do Agile”. When they hear “DevOps” or “Agile”, what they are hearing is is Let’s take the same people who can’t handle a planned release schedule or make whatever effort that they need to squeak by the Change Board and Release Management requirements, and give them unfettered access to Production. Clearly, I’m not paged often enough.

So what is one to do? How is one to maintain their sanity in the face of increasing job scope, increasing demand for access and velocity, and little hope for an effective new-hire count? Not to mention continuing to juggle the existing volume of requests, and continuing to grease the existing gears to keep the machine running.

Get Help

Please note that I’m not saying “just”. There’s nothing just about this situation; there is nothing simple about any of this, and Justice hasn’t been seen in a long time in an environment where this is the norm. Most of these changes are difficult. They will take work, and will require convincing other teams to join in your cause.

Admitting you have a Problem

The problem (probably) isn’t technical. It’s almost entirely social.

Because SysAdmins are typically responsible for the environment, the easiest way to assure that the state is stable is to lock everyone else from it. While this helped with the goal of “keeping out unexpected changes”, it had a number of side effects.

First, a kind of learned helplessness has set in. Your customers and teammates became so used to being “hands-off”, that they don’t have the wherewithal to meet reasonable expectations. Since they’re uncomfortable making any changes, all changes must be made by the SysAdmins. This leads to your time being taken by having to perform lots of low-value tasks.

Some teams settle on the pattern of “hands off Production, but you have access to Staging”, but this is fraught with peril. The most common problem that stems from this is “Configuration Drift.” Config Drift is when you have different settings in one environment (or server) than the others. When the cost to discover what Production looks like is high, it’s more likely that people will either use defaults, make assumptions, or use the same configs that they use in their IDEs. “Works on my machine”, indeed.

This is a problem well-solved by Configuration Management tools, but you still need to be willing to trust your peers and give them access. If you want to be part of the process of validating changes, you could put in place the structures that allow a pull-request and code-review workflow, something that your Software Engineering peers should be very accustomed to! Granting access to see the existing configs and the ability to propose changes also shares responsibility for your team’s environments and contributes to feelings of ownership. Denying colleagues the ability to effect necessary configuration changes contributes to the root problems of configuration drift and learned helplessness.

Stop Feeding the Machine

Your value is not in doing the work, but rather being able to make the decision to do the work.
I’ll be the first to say that “Automating ALL THE THINGS” is a flawed goal. At work, it’s usually said in the context of a Project, rather than part of a philosophy of Continual Improvement (Think Toyota Kata). You shouldn’t have to engage in an “Automation Project” to improve your environment. Build into your schedule time to solve one problem. Pick something that is rough, manual, and repeatable. Remove a small piece of friction. Move on to the next one. Hint: Logging into a server to make a configuration change should be a cue to implement configuration management!

While I agree that everything being automated, not everything should be automatic. Decision-making is complex, and attempting to codify all of the possible decision-making points is a fantastic way to make yourself insane. Not to mention that documenting your decision-making processes may be an unwanted look inside your brain. Caveat Implementor. (Or perhaps that’s just me…)

All of the units of work should be automated. But the decision to run the now-automated tasks can be left to a human. When you find that there is a correlation between steps, those pieces should be wrapped together. Automation isn’t a project into itself. It should be iterative. Pick something that’s painful. Make it a little smoother. Repeat. Ideally, you have time blocked out for Continuous Improvement. If not, create a meeting, or create a weekly project to do so. Review the issues that you’ve experienced lately, and pick something to make better. It might be worth making into a project, but it won’t be an ALL-THE-THINGS project. Create a scope of effort. Take the time to plan goals and requirements.

Whatever you don’t automate must be documented. Beyond the typical benefits of documentation, it also serves as “Functional Requirements” for someone else to pick up when they can help you with providing a solution. Try to recognize whether documenting or automating takes longer. Perhaps this piece of documentation will bet better served by “Executable Documentation” (i.e. code).

Clarify Your Role

You should attempt to pick apart the parts of your work, and attempt to describe them. One way to make this a fun exercise is to use other job titles to describe the work.

Are you an “Internet Plumber”? How much of your job could be described as “Spelunking” into the deep dark caverns of Legacy systems?

If you want, you could ascribe Superhero names to these parts of your work. The added bonus is that it not only describes a role, but also a demeanor associated with them. When ‘bad code’ makes it to Production, do you go “Wolverine” on that dev team?

Could you describe part of your role as “Production Customs Official”? Are you the gateway to Production? If so, are you actually equipped to do that? Here’s a quick test: When you say “no, that can’t go”, do you get overridden?

More importantly, is this what you want to do?

Prepwork

You will need to prepare for this. Most SysAdmin teams do not have a healthy relationship with the rest of the business. You will need to initiate the healing.

Take someone to lunch. Preferably someone who you don’t know well. Ask questions, and listen to the answers. It is not time to defend yourself or your team. It’s time to find out what the business needs from someone else’s perspective. Ask what they think that your team’s role is in toward achieving that success. Ask what they think your team does well, and where there are gaps between what you have now and excellence.

Speak their language

You probably recognize their words, but you need to go out of your way to speak them. To communicate your message, you will need meet them on their turf. This may seem terribly unfair - “Why can’t they meet me on my terms?!” - but I’m guessing that has not been working out well for you so far.

Not only do you need to use their language, but you need to communicate over their medium. And identifying who they are is step one in learning to speak it. It’s probably not IRC, and only writing it in email is a good way for it to be ignored.

If you’re speaking to management, be prepared to write a presentation. Executives especially like to see a slide-deck. It doesn’t have to be slick. It probably shouldn’t have sounds or much in the way of transitions, but a presentation can help to lay the groundwork for a conversation.

Discuss Scope, Staffing, and Priorities

Now that you have described your role, we also need to describe everything that you support.
What Products do you support? It’s entirely possible (likely, even) that the people and teams that you support don’t actually know what you’re responsible for. It could be argued that most of them shouldn’t need to know. But if you have been saying “no” to protect yourself, it’s a sign that you are significantly overextended. You need to have a real discussion with your leadership about your role, scope, and staffing.

In order to have this discussion, you need to prepare. You need to come up with a fairly comprehensive list of the products and teams that you support. This is a list of every team, and their products, the components and tasks that belong to you for each. Don’t forget all of the components that “nobody owns” but somehow people come to you to fix or implement (CI, SCM, Ticketing, Project, and Wiki tools seem to be common examples). Are you also responsible for Directory Services? Virtualization platform? Mail/Chat/Phones? Workstation Purchasing and provisioning? Printers? Do you manage the Storage, Networking, etc? Don’t be afraid of getting into details. It can help to provide clearly written potential impacts the company if some of these “hidden” services stop working? Your leadership might not know what LDAP or Directory Services are, but they’ll understand if nobody can log into their machines, they can’t pull information to build reports, and by-the-way nobody can deploy code because it relies on validating credentials…

What is most important to the company? What do you need to succeed? How much more staff do you need? What tooling or equipment would help you work more efficiently? Does code deploy even when it fails testing? How many outages have arisen due to this happening?

Demonstrate Cost and Value and Revisit Priorities

In order to have meaningful discussions with people in your company who aren’t necessarily technical, you need to be able to relate to a language that they speak. Regardless of team duties, the lingua franca of most teams is money. As Engineers, most of us prefer to think in terms of the tech itself, but in order to describe an impact, a unit of monetary value is a proxy for impact that most non-technical people can understand, even if they don’t grasp the details.

It is a helpful (if difficult and uncomfortable) habit to get into, but I encourage you to consider the components of cost that goes into every incident or task.

What is the cost of a main-site outage? How much revenue does this feature bring in? Why are you spending so much on infrastructure and effort to make that component Highly-Available? Why does it matter that you do that piece of maintenance? Show the negative value of doing things they way they are (Opportunity Cost), versus investing time to improve the automation around it. Describe how doing this maintenance work reduces your context switching, unplanned outages, and lost reputation of your company. Describe the benefit in increased visibility to the business, and Agency to be gained by your peers on other teams.

Why put in place these tools to let product teams self-serve? Describe that the features that the company’s teams spend so much time and effort (read: “money”) creating means nothing if those features aren’t available for customers to use. That having those features not available costs money in terms of feature billing, and reputation cost. If they claim that they’re doing Agile, but can’t do Continuous Delivery, they’re not really Agile, and the whole point of that framework is to improve delivery of value to the customer and the business!

Further, show how systems relate. It doesn’t have to be terribly detailed. Describe that the features that the customers use are reliant on x, y, and z components of infrastructure. Draw the lines from LDAP to storage to your CI tool to testing code to artifacts delivered to Production. Then show some of the other systems that have similar dependencies.

Once the picture emerges showing how everything is reliant on unexciting things like LDAP, your Storage cluster, and that janky collection of angry shell and perl scripts that keep everything working, realization will begin to dawn.

Congratulations, you’ve just effectively communicated Value.

Align Responsibility with Authority

Are you held responsible for apps written by other people? Who gets paged when “the app” goes down? How does that make sense?

Get Devs on-call for their apps. SysAdmins should be escalated to. Devs can triage and troubleshoot their own apps more readily than you can. They get to call in the cavalry when they get stuck. They don’t need to know everything about the systems, and they don’t need to resolve everything. When a fault occurs and they need help, they stay on the call, pairing with you as you diagnose, troubleshoot, and resolve. That way, they don’t need to escalate to you for that thing the next time it occurs, and can collaborate on automating a permanent fix.

When teams aren’t responsible for their products - When they aren’t paged when it fails - they are numb to the pain that they inflict. They’re not trying to cause pain; they just don’t feel it. It’s especially easy to argue this for teams that proclaim that they use Agile development methods: If they claim to want “continuous feedback”, there is nothing more visceral for providing feedback than the feeling of being awoken by a pager in the middle of the night. When the inevitable exclaimation comes that “we can’t interrupt our developers”, ask if it makes sense to interrupt someone else.
Even being aware of the pain (say, hearing how many times you were paged last night) can elicit sympathy, but that’s a far cry from the experience of being paged yourself.

Further, this is what that list of responsibilities is for. Asking each team to take responsilbity for their own products, you will still likely have a hefty list of services that you provide that you are on-call for. As these set in, point out the staffing numbers. This may be a matter of the places that I have worked, but I have never seen a Developer-to-SysAdmin ratio of less than 5-1. In most places it is much higher. By adding these teams to pager rotations, they drastically reduce the load on you. By not adding them to pager rotations, they are complicit in your burnout.

Stop saying “No”

SysAdmins have a reputation for saying “No”. The people who are asking are probably not trying to make your life worse; They’re probably just trying to get their work done. They might not know what their “simple request” involves, and that it probably isn’t necessary.

But by not having Responsibility aligned with Authority, you may have been stuck with the pain of other people’s wishes. You know that fulfilling their request will cause you pain, so understandably, you say “no”. What often happens next is that they escalate until they hit someone sufficiently important enough to override you.

This is the basis for why SysAdmins feel steamrolled by everyone else, and everyone else feels held hostage by SysAdmins.

But all hope is not lost.

Stop saying “No”.

“Yes, but …” is a very powerful thing.

“Yes, but …” can be used to get you help.

“Yes, I can set that up for you, but we don’t have capacity to run it for you.” What happened there? You agreed that the request is reasonable. You set expectations of the level of support that you can give. You left the requestor with several options to continue the conversation.

• They might have hiring reqs that they can’t fill. You can negotiate for some of them to go to your team, as you’re clearly understaffed.
• Some of their engineers may join your team as a lateral move. They’ll need mentorship and training, but this kind of cross-training is invaluable. It’s a force multiplier. It also sets precedent.
• They might take the responsibility for the Thing. They run it. They get paged for it. Of course you will probably have to be an escalation point to assist when it fails, but it’s their product. This again sets precedent.
  

Delegate

Most SysAdmins are stuck doing tasks that provide very little value because they restrict access to their peers. To my mind, there is one perfect example: “Playing Telephone”.

When I say “Playing Telephone”, I’m talking about the situation where someone (let’s say a Developer) wants logs from the application, but they don’t have access to get them. They request the logs from you. You fetch the log requested and provide it to them. “No, not that log, this log…” You fetch. “Hmm, I’m not seeing what I’m looking for, could you check in here for something that says something like this …?” And so on, and so on…

I don’t know what you’re hoping to prevent by restricting access, but if this scenario ever happens, you should know that you’re providing Negative Value. Again, let’s try to remember that your peers are not out to get you, and can probably be trusted to be reasonable humans if you meet them mid-way.

With that framework in mind, it’s time that you demonstrate some trust, and Delegate to them. Give them access. Your value is not in the logon credentials that you have, otherwise you’re just a poorly-implemented “Terminal-as-a-Service”.

Even better than giving access, is giving Tooling. Logging into a server should be an antipattern for most work! You need some better tooling. So, with the example of logging, let’s talk tooling.

Logging

First, logging into boxes to get logs is just dumb. Sure, you could wrap a tail command in a Rundeck job, but let’s Centralize those logs while we’re at it.

SysLog is better than nothing, but not by much. Shipping logs is easy, but consuming them as something useful is not. Batteries not included.

If your company wants to spend the money on Splunk, then encourage that. Splunk is a fantastic suite of tools, but I might wave you away from it if you’re not going to use it for everything. It’s going to be expensive, and if you’re not going to spend enough to use it for everything, there will be confusion as to what’s in there, and what’s stored elsewhere.

ELK (ElasticSearch + Logstash + Kibana, sometimes mistakenly simplified to “Logstash”), or “Cloudy Elk” / “ELK-as-a-Service” is a good middle-ground. ELK is Free (as-in-beer), and very featureful.

Take your Centralized logging of choice, and provide your customers with the url to the web interface. Send them links to the “How to use” docs, and get out of their way!

Terminal-as-a-Service

If someone asks you to “run this command for me”, you need to put a button on it.

You don’t need to RUN-ALL-THE-THINGS!

Rundeck is a fantastic tool to “Put a button on it”. Other people use their CI tools (like Jenkins or Bamboo) for this. My friend Jeremy Price gave an Ignite Talk at DevOpsDays NYC 2015 that describes this.

Personally I like Rundeck, because it’s pretty easy to make HA, tie it into LDAP for credentials, manage permissions, and by shipping it’s logs (see what I did there?), you get Auditing of who ran what and when!

If you have some data that Must be restricted, try to isolate those cases from the rest of your environment. You shouldn’t have to restrict Everything just because Something does need isolation.

Deploying Code. Yes, to Production

Why would you want to have to deploy other people’s code?! Do you really provide any value in that activity? If the deployment doesn’t go well, you’re launching another game of “Telephone”.
What if you make it easy for them to do it? Empower them with trust and tooling, making it easy to do the right thing! Give them tooling to see that the deploy succeeded! Logs are a start, but Metrics Dashboards that show changes in performance conditions and error rates will make it plain to see if a deployment was successful!

This Freedom doesn’t come free. Providing tooling doesn’t absolve the development teams of the need to communicate; in fact, it’s likely that they’ll have to communicate more. They will need to be watching those dashboards and logs to see for themselves the success of every deploy. They will also be more readily on-hand to help triage the inevitable instances when it doesn’t go swimmingly.

Us

I say “They” in this article a lot. And that is because, by default, most organizations that I have been a part of or heard stories of have had a strong component of “Us-Versus-Them.” It’s only natural for there to be an “Us” and a “Them”, but thinking in those terms should be a very short-term use of the language. Strive for the goal of a “We” in your interactions at work, and reinforce that language wherever possible. While it may not be My job to do “foo”, it is Our job to ensure the team and company is successful.

While that may sounds like some happy-go-lucky, tree-hugging, pop-psychological nonsense (and it is…:), the goal here is to get you, the beleaguered SysAdmin the help that you need, in order to improve the capabilites of the business.

Coda

There is so much more to this topic, particularly the shift away from a Systems team supporting a bunch of Project teams to a series of largely self-sustaining Product teams, but that will have to wait for another day.

The psychological damage done to SysAdmins by their peers can make us bitter and cynical. I encourage my people to try to see that “They” aren’t trying to make life difficult for you, but it’s very likely that Authority and Responsibility are misaligned. I likewise encourage my people to take steps to make their lives better. A ship’s course is changed in small degrees over time.

When someone says “DevOps Doesn’t Work”, they’re absolutely correct. DevOps is a concept, a philosophy, a professional movement based in trust and collaboration among teams, to align them to business needs. A concept doesn’t do work, and a philosophy does not meet goals - people do. I encourage you to seek out ways of working better with your fellow people.

Gratitude

I’d like to thank my friends for listening to me rant, and my editor Shaun Mouton for their help bringing this article together. I’d also like to thank the SysAdvent team for putting in the effort that keeps this fun tradition going.

Contact Me

If you wish to discuss with me further, please feel free to reach out to me. I am gwaldo on Twitter and Gmail/Hangouts, and seldom refuse hugs (or offers of beverage and company) at conferences. Death Threats and unpleasantness beyond the realm of constructive Criticism may be sent to:

Waldo
c/o FBI Headquarters 
935 Pennsylvania Avenue, NW
Washington, D.C.
20535-0001

Day 14 - Manage Your Mac Without Going Insane

Written by: Corey Quinn (@quinnypig)
Edited by: Ben Cotton (@funnelfiasco)

Like many people whose jobs include Linux administration, my corporate workstation is a Mac.

Like many people who work for large financial institutions, I don't do personal work on my corporate machine. So, I have a second Mac that lives at home.

This leads to an interesting pair of problems:

1. How do I keep preference files or other things I care about synced between these two machines without including data I don't want crossing the wall between personal and corporate systems?
2. How do I keep the current state of my machine backed up continuously so that if my computer gets hurled into the bay, I lose as little configuration data as possible?

The answer to both of these takes the form of a relatively small collection of open source tools, plus Dropbox. (Although you can easily substitute in Google Drive, iCloud, or, frankly, anything that you like that reliably syncs a local filesystem to a remote source, I'm using Dropbox in my examples for simplicity.)

Philosophically, I'm opposed to writing my own localized tools to solve global problems; therefore my solutions tend to involve stringing together freely available tools with (at most) a couple lines of shell to tie everything together. However, traditional configuration management tools tend to fall down for this use case. Puppet, Chef, or Salt are great for managing fleets of servers, but they require planning out what the finished product is going to look like. For a personal workstation, I often don’t know what tools I’m going to install until I’ve already installed them, so configuration management tools are forever left to play catch-up to what my current state happens to be.

The first thing I do is keep my dotfiles in a variety of different git repositories. I have separate git repositories for my vim, ssh, and git configurations and a few others, as well. This lets me be a bit more granular with respect to what data lives on different nodes-- for instance, I don’t want my work AWS credentials on my personal machine (and whether or not secrets belong in git repositories is an entirely separate conversation as well!). I tie them all together by managing them with vcsh. The specific details of exactly how vcsh works is out of scope for this article. That said, you can skip this step without seeing a meaningful difference for your environment.

Git based systems are all well and good (presuming you remember to push every time you update something!) for the traditional Unix style tools that store their configuration in static flat files, but in 2015 you can’t really assume that this maps to every application you’re using. “I store my Xcode configuration in git” is a recipe for tears before bedtime. Additionally, while git’s configuration lives in ~/.gitconfig, Skype’s configuration lives in a rats nest of a custom hierarchy under the relatively arcane ~/Library/Application Support/Skype/. This is where Mackup enters the picture.

What Mackup does falls squarely into the “so simple it’s genius” category. It copies the configuration of every application that it knows about (and that you have installed) to a location (by default your Dropbox folder, under a “Mackup” directory, but this is easily changed) and replaces the old location with a symlink to the configuration’s new home. As a result, I can make a configuration change to an application on my work machine, and when I start that application at home, that configuration change magically propagates.

For those of us who work in security conscious environments, Mackup features both an application whitelist and an application blacklist, to your preference. I’m thrilled with using Dropbox to store my iTerm2 configs, but I’m not quite so content with storing my GPG keys there. There are also a couple of applications I want different configs on for different machines, so I can exclude those as well. Lastly, Mackup also features an “uninstall” command, so if you try it and discover it’s not for you, it sets things right as if it had never been used.

This handily solves the configuration question, but what about packages themselves? In case you’ve been living in a cave for the past few years, the Mac App Store is a burning dumpster fire of a neglected ecosystem; there’s no way to hook into it via an API, so you’re stuck with manual installation there. As a result, I tend to mostly ignore it in favor of homebrew (for command line utilities on the Mac) and homebrew-cask (like homebrew, except for graphical applications; it functions via installing packages into /opt/homebrew-cask/ and symlinking them into ~/Applications/ for you). Then I go about my day— “Sorry to do this to you, but you’re going to need to brew install gnu-cobol to get this node project to run—“ without caring about updating a manifest of packages that are installed. Then, either manually or via a scheduled task (launchd if you want to drink the OS X kool-aid, cron if you don’t and care to install it) launch a brew-bundle one-line task that dumps out the list of currently installed applications, and copies it to Dropbox:

brew bundle dump --force —file=~/Dropbox/Brewfile

I don’t tend to sync this between machines, as the list of packages I want installed on both differs, but it does make “replacing a computer” a trivial task— I simply install Homebrew and Dropbox, let it sync, then cd ~/Dropbox; brew bundle; mackup restore before stepping out for coffee. I’ll come back to all of my applications installed, configured, and waiting for me.

Note that this entire approach does have its limits. The idea of partial synchronization doesn’t really work here-- you can get granular down to the level of individual applications, but getting more fine-grained than that tends to not work. You can’t, for instance, sync some of your Firefox or Chrome bookmarks, but not include your extensions; at this point, Mackup is an all-or-nothing solution in the context of specific applications. Likewise, if you decide to share brew bundle files between machines, you’re going to have a difficult time reconciling different package build flags (which brew bundle captures!) between machines.

It’s also worth pointing out that this particular approach may not work for everyone. There’s a perfectly valid school of thought that says that you should opt for keeping things as close to pristine as possible-- and there’s a benefit to that. That said, for some use cases this workflow works well for me-- and I hope it’ll get you thinking about ways you can improve your own.

Day 13 - An introduction to OpenShift 3

Written by: Tobias Brunner (@tobruzh)
Edited by: Nick Silkey (@filler)

Over the last year there has been a lot of buzz around open-source Platform-as-a-Service (PaaS) tools. This blog will give you an overview of this topic and some deeper insight into Red Hat OpenShift 3. It will talk about the kind of PaaS tools you install on your own infrastructure - be it physical, virtual or anywhere in the cloud. It does not cover installation atop Heroku or similar services, which are hosted PaaS solutions.

But what exactly is meant by PaaS?

When we talk about PaaS, we mean a collection of services and functions which serve to orchestrate the processes involved with building software up to running it in production. All software tasks are completely automated: building, testing, deploying, running, monitoring, and more. Software will be deployed by a mechanism similar to "git push" to a remote source code repository which triggers all the automatic processes around it. While every PaaS platform behaves a bit differently, in the end they all do the same function of running applications.

Regarding running an application within a PaaS, the application should optimally be designed with some best practices in mind. A good guideline which incorporates these practices is commonly known as the The Twelve-Factor App.

Short overview of Open Source PaaS platforms

There are a lot of Open Source PaaS tools in this space as of late

• Dokku: http://progrium.viewdocs.io/dokku/ A simple, small PaaS running on one host. Uses Docker and Nginx as the most important building blocks. It is written in Bash and uses Buildpacks to build an application specific Docker container.

• Deis: http://deis.io/ The big sister of Dokku. Building blocks used in Deis include CoreOS, Ceph, Docker, and a pluggable scheduler (Fleet by default, however Kubernetes will be available in the future). Buildpacks are used for creating the runtime containers. At least three servers are needed to effectively run Deis.

• Tsuru: https://tsuru.io/ Similar to Deis, but says it also supports non-12-Factor apps. There is even a possibility to manage VMs, not only containers. This, too, uses Docker as building block. Other components like scheduler are coming from the Tsuru project.

• Flynn: https://flynn.io/ Also similar to the two tools above. It also uses Docker as backend, but uses many project specific helper services. At the moment, only PostgreSQL is supported as a datastore for applications.

• Apache Stratos: http://stratos.apache.org/ This is more of a framework than just a "simple" platform. It is highly multi-tenant enabled and provides a lot of customization features. The architecture is very complex and has a lot of moving parts. Supports Docker and Kubernetes.

• Cloud Foundry: https://www.cloudfoundry.org/ One of the biggest player besides OpenShift in this market. It provides a platform for running applications and is used widely in the industry. Has a steep learning curve and is not easily installed,configured, or operated

• OpenShift: http://www.openshift.org/ Started in 2011 as a new PaaS platform using it’s own project specific technologies, has been completely rewritten in v3 using Docker and Kubernetes as the underlying building blocks.

OpenShift

There are some compelling reasons to look at OpenShift:

• Usage of existing technology Kubernetes and Docker are supported by big communities and are good selections to serve as central components upon which OpenShift is built. OpenShift "just" adds the functionality to make the whole platform production-ready. I also like the approach of Red Hat to completely refactor OpenShift V2 to V3, taking into account what they have learned from older versions and not just simply trying to improve the old code base on top of Kubernetes and Docker.

• Open development Development happens publically on GitHub: https://github.com/openshift/origin. OpenShift Origin is"the upstream open source version of Red Hat's distributed application system, OpenShift" per Red Hat.

• Enterprise support available Many enterprises want to or need to have support contracts available for the software which they run their business upon. This is completely possible using the OpenShift Enterprise subscription which gets you commercial support from Red Hat.

• Excellent documentation The documentation at https://docs.openshift.org/latest/welcome/index.html is very well structured, allowing for rapid identification of a topic youre seeking.

• My favorite functions There are some functions which I like the most on OpenShift:

  • Application templates: Define all components and variables to run your application and then instantiate it very quickly multiple times with different parameters.

  • CLI: The CLI tool "oc" is very well structured and you get it very quickly how to work with it. It also has very good help instructions integrated, including some good examples.

  • Scaling: Scaling an application just takes one command to start new instances and automatically add them to the load balancer.

So why not chose Cloud Foundry? At the end of the day, everyone has their favourite tool for their own reasons. I personally found the learning curve for Cloud Foundry to be too steep. I didn’t manage to get an installation up and running with success. Also I had lots of trouble to understand the things around BOSH, a Cloud Foundry-specific configuration management implementation.

Quick Insight into OpenShift 3 - What does it look like?

OpenShift consists of these main components:

• Master The API and Kubernetes Master / Scheduler

• Nodes Runs pods, including the workload

• Registry Hosts docker images

• Services / Router Takes client requests and routes them to backend application containers. In a default configuration it’s a HAProxy load balancer, automatically managed by OpenShift.

For a deeper insight, consult the OpenShift Architecture

OpenShifts core is Kubernetes with additional functionality for application building and deployment made available to users and operators. So it’s very important to understand the concepts and the architecture of Kubernetes. Consult the official Kubernetes documentation to learn more: http://kubernetes.io/v1.1/

Communication between clients and the OpenShift control plane happens over REST APIs. The oc application, available via the OpenShift command-line client, gives access to all the frequently-used actions like deploying applications, creating projects, viewing statuses, etc.

Every API call must be authenticated. This authentication is also used to check if you’re authorized to execute the action. This authentication component allows for OpenShift to support multi-tenancy. Every OpenShift project has it’s own access rules. Projects are separate from each other. On the network side, they can be strictly isolated from each other via the ovs-multitenant network plugin. This means many users can share a single OpenShift platform without interfering each other.

Administrative tasks are done using oadm within the OpenShift command-line client. Example tasks involve operations such as deploying a router or a registry.

There is a helpful web interface which communicates to the master via the API and provides a graphical visualization of the cluster’s state.

Most of the tasks from the CLI can also be accomplished via the GUI.

To better understand OpenShift and its core-component Kubernetes, it’s important to understand some key terms:

• Pod "In Kubernetes, all containers run inside pods. A pod can host a single container, or multiple cooperating containers". Roughly translated, this means that containers share the same IP address, the same Docker volumes and will run always on the same host.

• Service A pod can offer services which can be consumed by other pods. Services are addressed by their name. This means for example a pod provides a HTTP service on the name "backend" on port 80. Another pod can now access this HTTP service by just addressing the namespace of “backend”. Services can be exposed externally from OpenShift using an OpenShift router.

• Replication Controller A replication controller takes care of starting up a pod and keeping it running in the event of a node failure or any other disruptive event which could take a pod down. It is also responsible for creating replication pods in an effort to horizontally scale the lone pod.

Quickstart 3-node Origin Cluster on CentOS

There are a number of good instructions how to install OpenShift. This section will just give a very quick introduction to installing a 3-node (1 master, 2 nodes) OpenShift Origin cluster on CentOS 7.

If you want to know more details the OpenShift documentation at Installation and Configuration is quite helpful. The whole installation process is automated using Ansible, all playbooks are provided by the OpenShift project on Github. You also have the option to run an OpenShift master inside a Docker container, as a single binary or installing it from source. However the Ansible playbook is quite helpful for getting started.

Pre-requisites for this setup

1. Three CentOS 7 64-Bit VMs prepared, each having 4+GB RAM, 2 vCPUs, 2 Disks attached (one for the OS, one for Docker storage). The master VM should have a third disk attached for the shared storage (exported by NFS in this example), mounted under /srv/data.

2. Make sure you can access all the nodes from the master using SSH without a password. This is typically accomplished using ssh keys. The above user also needs sudo rights, typically via the NOPASSWD option in typical Ansible fashion.

3. A wildcard DNS entry pointing to the IP address of the master. It is at this master where routers will run to allow for external clients to request application resources running within OpenShift.

4. Read the following page carefully to make sure your VMs fit the needs of OpenShift: Installation Pre-requisites Pay special attention to the Host Preparation section.

5. Ensure you have pyOpenSSL installed on the master node as it was a missing dependency during the time of writing this article. You can install it via yum -y install pyOpenSSL.

6. Run all the following steps as your login user as opposed to root on the master node.

Setup OpenShift using Ansible

1. Put the following lines into /etc/ansible/hosts. Update the values to fit your environment (hostnames):

   [OSEv3:children]
   masters
   nodes
   [OSEv3:vars]
   ansible_ssh_user=myuser
   ansible_sudo=true
   deployment_type=origin
   osm_default_subdomain=apps.example.com
   osm_default_node_selector='region=primary'
   openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}]
   [masters]
   originmaster.example.com
   [nodes]
   originmaster.example.com openshift_node_labels="{'region': 'infra', 'zone': 'dc1'}" openshift_schedulable=true
   originnode1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'dc1'}" openshift_schedulable=true
   originnode2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'dc1'}" openshift_schedulable=true

2. Run Ansible: ansible-playbook playbooks/byo/config.yml

3. Check that everything went well via the OpenShift client. Your output should show 3 nodes with STATUS ready: oc get nodes

4. Add a user to OpenShift: sudo htpasswd /etc/origin/htpasswd myuser

5. Deploy the registry (additional details at Deploy Registry):

   1. sudo htpasswd /etc/origin/htpasswd reguser

   2. oadm policy add-role-to-user system:registry reguser

   3. sudo oadm registry --mount-host=/srv/data/registry --credentials=/etc/origin/master/openshift-registry.kubeconfig --service-account=registry

6. Deploy the router (additional details at Deploy Router):

   1. sudo oadm router --credentials=/etc/origin/master/openshift-router.kubeconfig --service-account=router

Add persistent storage

After these steps, one important piece is missing: Persistent storage. Running any application which stores application data will lose its data after migration to another OpenShift node after restarting, redeployment, and so on. Therefore we should add shared NFS storage. In our example, we will make this available by the master:

1. Add the following line to /etc/exports: /srv/data/pv001 *(rw,sync,root_squash)

2. Export the directory: sudo exportfs -a

3. On all cluster nodes including the master, alter the SELinux policy in order to enable the usage of NFS: sudo setsebool -P virt_use_nfs 1

4. Create a file called pv001.yaml with the following content:: apiVersion: "v1" kind: "PersistentVolume" metadata: name: "pv001" spec: capacity: storage: "30Gi" accessModes:

   • "ReadWriteOnce" nfs: path: "/srv/data/pv001" server: "originmaster.example.com" persistentVolumeReclaimPolicy: "Recycle"

5. Create the definition from the template you just created: oc create -f pv001.yaml

6. Check the definition: oc get pv

You now have a 3-node OpenShift Origin cluster including persistent storage, ready for your applications!

Please note: This is a very minimal setup. There are several things to do before running in production. F.e. you could add some constraints to make sure that the router and registry only run on the master and all applications on the nodes. Other things to include into production deployment considerations: storage capacity, network segmentation, security, accounts. Many of these topics are discussed in the official OpenShift documentation.

Deploy applications

Now that the 3-node cluster is up and running, we want to deploy apps! Of course, that’s the reason we created it in the first place, right? OpenShift comes with a bunch of application templates which can be used right away. Let’s start with a very simple Django with PostgreSQL example.

Before you begin, fork the original GitHub project of the Django example application to your personal GitHub account so that you’re able to make changes and trigger a rebuild. The intent here allows you to control webhook configurations for your fork to trigger code updates within OpenShift. Once done, then you can proceed to create the project within OpenShift.

1. Login to https://originmaster.example.com:8443/ with the user created in step 4 above.

2. Click on "New Project" and fill in all the fields. Name it myproject. After clicking on “Create” you’re directed to the overview of available templates.

3. Create a new app by choosing the django-psql-example template

4. Insert the repository URL of your forked Django example application into the field SOURCE_REPOSITORY_URL. All other fields can use their default value. By clicking on "Create" all required processes are started in the background automatically.

5. The next page gives you some important hints. To have the complete magic: To automate the delivery of new code via a deployment without human intervention, configure the displayed webhook URL in your Github project. Now every time you push code to your git remote on GitHub, OpenShift gets notified and will rebuild and redeploy your app without effort spent by a human. Please note that you probably need to disable SSL verification as by default a self-signed certificate is used which would fail verification

6. Watch your new app being built and deployed on the overview page. As soon as this is finished, the app is reachable at http://django-psql-example-myproject.apps.example.com/

7. To get the feeling of the full automation: Change some code in the forked project and push it to GitHub. After a few seconds, reload the page and you should see your changes active.

Scaling

One of the most exciting features in OpenShift is scaling. Let’s say the above Django application is an online shop and you create some advertisement which will lead into more page views. Now you want to make sure that your online shop is available and responsive during this time. Simple as that:

oc get rc

oc scale rc django-psql-example-2 --replicas=4

oc get rc

The replication controller (rc) takes care of creating more Django backends on your behalf. Other components will make sure that these new backends are added to the router as load balancing backends. The replication controller also makes sure that there are always 4 replicas running, even if a host fails and there are enough hosts available on the cluster to run the workload on.

Scaling down is just as easy as scaling up, just adjust the --replicas= parameter accordingly.

Another application

Now that we’ve deployed and scaled a default app, we want to deploy a more customized app. Let’s use Drupal 8 as an example.

Drupal is a PHP application which uses MySQL by default, so we need to use the matching environment for this. Set it up via the following command:

oc new-project drupal8

oc new-app php~https://github.com/tobru/drupal-openshift.git#openshift \

mysql --group=php+mysql -e MYSQL_USER=drupal -e \

MYSQL_PASSWORD=drupalPW -e MYSQL_DATABASE=drupal

This long command needs a bit of explanation:

• oc: Name of the command line OpenShift client

• new-app: Subcommand to create a new application in OpenShift

• php~: Specifies the build container to use (if not provided the S2I process tries to find out the correct one to use)

• https://github.com/tobru/drupal-openshift.git: Path to the git repository with the app source to clone and use

• #openshift: Git reference to use. In this case the openshift branch

• mysql: Another application to use, in this case a MySQL container

• –group=php+mysql: Group the two applications together in a Pod

• -e: Environment variables to use during container runtime

The new-app command uses some special sauce to determine what build strategy it should use. If there is a Dockerfile present in the root of the cloned repository, it will build a Docker image accordingly. If there is no Dockerfile available, it tries to find out the project’s language and uses the Source-to-Image (S2I) process to build a matching Docker image containing both the application runtime and the application code.

In the example above we specify to use a PHP application container and a MySQL container to group them together in a pod. To successfully execute the MySQL container a few environment variables are needed.

After the application has been built, the service can be exposed to the world:

oc expose service drupal-openshift --hostname='drupal8.apps.example.com'

As the application repository is not perfect and the PHP image not configured correctly for Drupal (perhaps we hit the issue #73!), we need to run a small command inside the pod to complete the Drupal installation. Substitute ID with the ID of your running pod which is visible via oc get pods):

oc exec drupal-openshift-1-ID php composer.phar install

Now navigate to the URL drupal8.apps.example.com and run the installation wizard. You’ll want to set the DB host to "127.0.0.1" as “localhost” doesn’t work as you might expect.

At the end there is still a lot to do to get a "perfect" production-grade Drupal instance up and running. For example it is probably not a good idea to run the application and database in the same pod because it makes scaling difficult. Scaling a pod up scales all Docker images contained within the pod. This means that the database image also needs to know how to scale, which is not the default case. In it’s heart, scaling means just spinning up another instance of a Docker image, but that does not mean that the user generated data is automatically available to the additional running Docker images. This needs some more effort put in. The best thing here would be to have different pods for the different software types: One pod for the Drupal instance and one pod for the database so that they can be scaled independently and the task of replicating user generated data can be done tailored to the softwares need (which is of course differently between a web server and a database server).

Also there is persistent storage missing in this example. Every uploaded file or other working files will be lost when the pod is restarted. If you want more information on how to most effectively add persistent storage, here is a very good blog post describing it in detail: OpenShift v3: Unlocking the Power of Persistent Storage.

Helpful commands

When working with OpenShift, it’s useful to have some commands ready that help getting information or show what is going on.

OpenShift CLI: oc

The OpenShift CLI "oc" can be installed on your computer, see Get Started with the CLI. Before issuing an oc command, you must login to the OpenShift master with oc login. It will ask for an URL to the master (if started for the first time) and for a username and password.

• oc whoami
  If you can’t remember who you are, this tells it to you.

• oc project $NAME
  Shows the currently active project to which all commands are run against. If a project ame is added to the command then the currently active project changes.

• oc get projects
  Displays a list of projects to which the current user has access to

• oc status
  Status overview of the current project

• oc describe $TYPE $NAME
  Detailed information about an object.oc describe pod drupal-openshift-1-m3uvx

• oc get event
  Shows all events in the current project. Very useful for finding out what happened.

• oc logs [-f] $PODNAME
  Show the logs of a running pod. With -f it tails the log much like tail -f.

• oc get pod [-w]
  List of pods in the current project. With -w it shows changes in pods. Note: watch oc get pod is a helpful way to watch for pod changes

• oc rsh $PODNAME
  Start a remote shell in the running pod to execute commands

• oc exec $PODNAME $COMMAND
  Execute a command in the running pod. The command’s output is sent to your shell.

• oc delete events --all
  Cleanup all events. Useful if there are a lot of old events. Events are information about what is going on on the API objects and what problems exist (if there are any).

• oc get builds
  List of builds. A build is a process of creating runnable images to be used on OpenShift.

• oc logs build/$BUILDID
  Build log of the build with the id "buildid". This corresponds to the list of builds which are displayed with the command above.

Commands to run on the OpenShift server

The OpenShift processes are managed by systemd. All logs are written to the systemd journal. So the easiest way to get information about what is going on it to query the system journal:

• sudo journalctl -f -l -u docker
  System logs of Docker

• sudo journalctl -f -l -u origin-master
  System logs of Origin Master

• sudo journalctl -f -l -u origin-node
  System logs of Origin Node

Some more interesting details

Router

The default router runs HAproxy which is dynamically reconfigured should new services be requested to be routed by the OpenShift client. It also exposes its statistics on the router's IP address over TCP port 1936. The password to retrieve this is shown during the application’s deployment or it can be retrieved by running oc exec router-1- cat /var/lib/haproxy/conf/haproxy.config | less. Look for a line beginning with "stats auth".

Note: for some reasons an iptables rule was missing on my master node preventing my getting at the router statistics. I added one manually to overcome via sudo iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 1936 -j ACCEPT

Logging

OpenShift brings a modified ELK (Elasticsearch-Logstash-Kibana) stack called EFK: Elasticsearch-FluentD-Kibana. Deployment is done using some templates and the OpenShift workflows. Detailed instructions can be found at Aggregating Container Logs in the OpenShift documentation. When correctly installed and configured as described under the above link, it integrates nicely with the web interface. These steps are not part of the Ansible playbooks and need to be carried out manually.

Metrics

Kubernetes Kubelets gather metrics from running pods. To collect all these metrics, the metric component needs to be deployed similar to the logging component via OpenShift workflows. This is also very well documented at Enabling Cluster Metrics in the OpenShift documentation. It integrates into the web interface fairly seamlessly..

Failover IPs

Creating a highly available service most of the time involves having IP addresses be made highly available. If you want to have a HA router, there is the concept of IP failover available within OpenShift. This means you configure an IP address as a failover address and attach it to a service. Under the hood, keepalived now keeps track of this IP and makes it highly available using the VRRP protocol.

IPv6

I couldn’t find much information about the state of IPv6 in OpenShift. But it seems problematic right now, as far is I can see. Docker is supporting IPv6 but Kubernetes seems to lack functionality: Kubernetes issue #1443

Lessons learned so far

During the last few months while making myself familiar with OpenShift I’ve learned that the following points are very important in understanding OpenShift:

• Learn Kubernetes This is the main building block in OpenShift, so a good knowledge is necessary to get around the concepts of OpenShift.

• Learn Docker As this is the second main building block of OpenShift, it’s also important to know how it works and what concepts it follows.

• Learn Twelve-Factor To get the most out of a PaaS, deployed applications should closely follow the Twelve-Factor document.

Some other points I learned during experimenting with OpenShift:

• The Ansible playbook is a fast moving target. Most of the time the paths written in the documentation don’t match the Ansible code sadly. Also some things didn’t work well, for example upgrading from Origin 1.0.8 to Origin 1.1.0.

• OpenShift heavily depends on some features, such as SELinux, which are by default only present on Red Hat-based Linux distributions. This makes it hard to go about getting OpenShift working on Ubuntu-based Linux distributions without things quickly becoming a yak shaving exercise. In theory it should be possible to run OpenShift on all distributions supporting Docker and Kubernetes, but as always the devil lies in the details.

• Proper preparation is the key to success. As OpenShift is a complex system, preparation helps to get a working system. This is not only preparing VMs and IP addresses, but also preparing knowledge, like learning how everything works and try it out in a test system.

Some more technical learnings:

• When redeploying the registry the master needs to be restarted, as the master caches the registry service IP. This is possible via: sudo systemctl restart origin-master

• Before running applications on OpenShift it makes sense to run the diagnostics tool using: sudo openshift ex diagnostics

Automate all the things!

The friendly people at Puzzle are working on a Puppet module to make the OpenShift installation and configuration as automated as possible. Internally it calls Ansible to do all the lifting. While it doesn’t make sense to re-implement everything in Puppet, the module helps with manual tasks that have to be carried out after having OpenShift installed by Ansible. For an already existing Puppet environment, this is great to get OpenShift integrated.

You can find the source on GitHub and help to make it even better: puzzle/puppet-openshift3.

Conclusion and Thanks

This blog post only touches the tip of the iceberg regarding OpenShift and its capabilities as an on-premise PaaS. It would need many books to cover all of this very exciting technology. For me it’s one of the most thrilling piece of technology since many years and I’m sure it will have a bright future!

If you are interested in running OpenShift in production, I suggest doing a proof-of-concept fairly early. Be prepared to read and plan a lot. Many important topics such as monitoring, security, backup were not covered here and are important topics for a production-ready PaaS

For those wanting additional reading materials on OpenShift, here are some additional links with lots of information:

• https://docs.openshift.org/

• http://www.openshift.org/

• http://lists.openshift.redhat.com/openshiftmm/listinfo

• https://blog.openshift.com/

  • https://blog.openshift.com/author/benparees/

• https://www.openshift.tv/

• https://github.com/openshift/training

• https://github.com/openshift/origin/tree/master/examples

I want to thank everyone who helped me with this blog, especially my workmates at VSHN (pronounced like vision / vĭzh'ən) and our friends at Puzzle ITC. And another special thank goes to Nick Silkey (@filler) for being the editor of this article.

If you have any comments, suggestions or if you want to discuss more about OpenShift and the PaaS topic, you can find me on Twitter @tobruzh or on my personal techblog called tobrunet.ch.

Day 12 - Introduction to Nomad

Written by: Patrick O'Connor (@dontrebootme)
Edited by: Justin Garrison (@rothgar)

Introducing Cluster Schedulers

Are you still running all of your containers with a single DOCKER_HOST? Have you managed to spread the load between two or three hosts but manually call docker run commands on each host to bootstrap your container infrastructure? If so, you should consider using a container scheduler instead.

Fleet, Kubernetes, Mesos, Nomad, Rancher, and Swarm are probably names you've heard of recently but are you familiar with what they actually do? Schedulers provide many benefits in a containerized environment. They are the next big step once you've played with a local development environment or when you are going to deploy your CI/CD pipeline. Container schedulers vary in their features and implementation but some of the core principles are the same.

• Pool resources from a group of hosts into their components (CPU/RAM) and make them available for consumption. Also make sure the resources don't become exhausted via over-provisioning or host failure.
• Service supervision provides a service load balancer/entry point and make sure the service remains running.
• Scaling functionality scales a service (automatic or manually) by allowing an operator to create more or fewer instances.
• System metadata provides stats about running instances, scheduling, and container health.

Running a scheduler has obvious benefits over running containers by hand, but to run some of these schedulers requires a long list of dependencies that are not trivial to set up and maintain. Enter Hashicorp's Nomad, from the team that brought you Consul, Vagrant, Packer, Terraform, Vault, and recently Otto. Nomad attempts to give you all of the container scheduler benefits without any of the complexity. It is a single binary and optional config file making it simple to deploy and get started. You don't need to make any changes to your existing containers and can get started with just a few commands.

Understanding Nomad

Nomad is built to be operationally simple. So simple that we can easily create an example cluster to play with on your local machine.

We are going to leverage Vagrant to automatically provision multiple machines that we can use to demonstrate container scheduling with Nomad. These machines will be: * Nomad server (1) * Consul server (1) * Docker host (3)

We will explain more about what each VM will do as we go. To get started, perform a git clone on nomad-intro. Make sure you have Vagrant installed.

git clone https://github.com/dontrebootme/nomad-intro
cd nomad-intro
vagrant up

This process starts and configures a Consul server, Nomad server, and some Docker hosts running Nomad and Consul agent. Once vagrant is done provisioning the VMs run vagrant status to confirm all instances are running:

➜  nomad-intro (master) ✔
 vagrant status
Current machine states:

consul                    running (virtualbox)
nomad                     running (virtualbox)
docker1                   running (virtualbox)
docker2                   running (virtualbox)
docker3                   running (virtualbox)

Nomad Server

Nomad running in server mode acts as the manager for the agents and applications in your cluster. A normal production cluster would have multiple Nomad servers for redundancy and failover. We describe how we want Nomad to run our applications by defining jobs. I've included a sample job with a task that launches a web server in the nomad-intro repository. The .nomad file defines a job, task group, and task. A job file will only ever describe a single job, but can have multiple tasks. Additional definitions in the job definition include data such as datacenter, region, instance count, update policy, resources allocated, networking speed, port allocations, and health checks.

In addition to nomad running the server, it also is the same binary used to interact with the cluster. To demonstrate how we schedule a job with nomad, take a look at the provided microbot.nomad file. You can run the microbot service on your cluster by issuing:

vagrant ssh nomad
nomad run /vagrant/microbot.nomad

Nomad will recieve the job definition and act on the request by scheduling the job on the agent nodes. You can view the status of the deployment with nomad status microbot.

Nomad Agent

Nomad running in agent mode will receive requests for tasks from the server and, if possible, act on those requests. In the example above we asked Nomad to start 9 instances of the microbot task which in this example are webservers running in a Docker container as defined by the job definition. We asked Nomad to allocate ports for the containers we launched and monitor the health of those services to act on them should the health check ever fail.

Consul

Consul is typically seen as two components, service discovery and distributed key value storage. We will focus on the service discovery portion of Consul for an automated way to discover and query our servers that may exist around our environment. Consul works well with Nomad, not surprisingly, because Nomad can automatically inform Consul of all running services, their host and port, and the health of the services.

With Nomad and Consul in sync, we can automate other systems such as the load balancer to automatically update when containers move or when containers are created and destroyed with scaling.

Further Experimentation:

Now that we've covered Nomad server, agent, and how we can leverage Consul for service discovery, let's do some further demonstrations of interacting with Nomad for tasks such as:

• Scale the microbot service
• Rolling updates
• System jobs
• Query Consul

Scale the Microbot Service

Let's demonstrate the ability to use Nomad to scale a service. Open the microbot.nomad file with your favorite text editor and change the instance value from 9 to 50.

...
group "webs" {
  # We want 9 web servers initially
  count = 50

  task "microbot" {
    driver = "docker"
...

Once you're done, tell Nomad to about our changes by running:

vagrant ssh nomad
nomad run /vagrant/microbot.nomad

Nomad will look at the new job definition and make sure your cluster count matches what is already running. In this case it will scale the service up with an additional 41 containers.

Rolling Updates

To demonstrate a rolling update, let's look at our job file microbot.nomad once again.

...
# Rolling updates should follow this policy
update {
  stagger = "10s"
  max_parallel = 5
}
...

stagger defines the time between new container deployments and max_parallel sets how many containers can be stopped and started in parallel.

We can push a new version of our container out by changing the version of our microbot container image from v1 to v2 in the microbot.nomad file.

task "microbot" {
  driver = "docker"
  config {
    image = "dontrebootme/microbot:v2"
  }
  service {
    port = "http"

Once you push the new application with nomad run /vagrant/microbot.nomad the containers will be updated, 5 at a time with 10 seconds between each batch.

More information about updates are available via the Nomad documentation

System Jobs

Nomad has three types of jobs: service, batch, and system. Our previous example used a service job which is intended for long running tasks. Let's schedule a new job of type = "system". If you take a look at cadvisor.nomad, you'll see an example of a system job. System jobs are great for deploying services/tools that you expect to be on every host. You would usually use this type of job for logging, monitoring, and possibly a local docker registry. In this example we'll use the metrics service cAdvisor. Let's deploy this on every Nomad client/Docker host by issuing our nomad run with this new job definition:

vagrant ssh nomad
nomad run /vagrant/cadvisor.nomad

To verify that it is running on all hosts run:

vagrant ssh nomad
nomad status cadvisor

The cadvisor job should be scheduled, one per host, on your cluster.

Query Consul

One of the features of Nomad is that it has native integration with Consul. We've covered the relationship of Nomad and Consul above, but we can take a look at Consul and verify that it is indeed receiving all the information about our services by using Consul data in two ways, via an API call, or via the web UI.

To ask Consul for all running services in our cluster:

# List running services
curl -s http://10.7.0.15:8500/v1/catalog/services
# List information about our microbot service
curl -s http://10.7.0.15:8500/v1/catalog/service/microbot-web-microbot

Or we can visit the web UI by browsing to http://10.7.0.15:8500/ui/

Spin Down and Clean Up

When you're done, you can shut down the cluster using

vagrant halt

# And free up some disk space using
vagrant destroy -f

# If you want to change any of the configuration/scripts run
vagrant provision

Conclusion

There are numerous options for container cluster schedulers and the list isn't getting shorter. Nomad's approach to keep things operationally simple with less infrastructure needs is a relief when some schedulers seem overly complex. This model allows for fast automation and lean system requirements. We've learned how we can leverage Nomad to distribute a task around multiple hosts, scale the services, and deploy updates while letting the cluster scheduler handle the placement, supervision, and rolling updates. If you're ready to take your container infrastructure to the next level but are looking for a simple scheduler to deploy and manage then Nomad may be the container scheduler you've been waiting for.