Day 9 - One year of PowerShell or DevOps in the Microsoft world...

Written by: Dominique Broeglin (@dbroeglin)
Edited by: Matt Stratton (@mattstratton)

Holidays are a time for reflection and giving back. This year, I thought I would reflect on my ongoing Linux to Microsoft transition and give back a few tidbits I learned about DevOps in the Microsoft world. This article is meant to illustrate a journey from Linux to Microsoft in hopes that the “from the trenches” perspective will help others attempting the same transition.

Let’s start with a quick background to set the stage.

Background

After an education in both software engineering and distributed systems, I worked for several IT companies, small and big, in a variety of domains such as telcos, education, justice, transportation, luxury goods, or the press. I was lucky enough to have the opportunity to work with bright people very early in my career. I’m very grateful to all of them, as I learned a lot from them and was spared quite a few mistakes. One of the subjects I was introduced to early is automation, both through my co-workers and through seminal books like The Pragmatic Programmer: From Journeyman to Master and its sequel Pragmatic Project Automation. This pushed me towards automated configuration management in the early days.

If memory serves, I discovered Linux somewhere around 1994 and was instantly hooked. I still recall that first Slackware distribution with its 50-something floppy disks. For twenty years, I’ve worked mainly on Linux and Unices, trying studiously to avoid any Microsoft product. Some people may even recall me swearing that I would never work on Windows. However, a few years ago Microsoft changed its approach, and I was, coincidentally, presented opportunities in Microsoft environments. As was once said by John H. Patterson, “Only fools and dead men don’t change their minds.” So I went on and changed mine, one step at a time…

Armed with almost 20 years of Linux experience and ten years of automated configuration management, I jumped into Microsoft automated configuration management.

Step 1: Develop on Windows, Deploy on Linux

The first step was a Java project that was starting. In that early design and setup phase, only a few people were involved. I was brought in to help with Java architecture and software factory setup. Having set up quite a few factories already that part was not new to me. However, we would have to onboard a sizable number of people in a small amount of time. That would be a challenge. Because, at the time it took quite a bit of time to setup a working environment for a developer, tester, architect or DBA.

During previous projects, I had used Vagrant to setup development environments that could be shared easily. Vagrant is a wrapper around a virtualization tool (at the time VirtualBox, but in the meantime, Hyper-V and VMWare were added too) that helps in setting up a virtual machine. It starts with a fixed base image, but allows for additional setup instructions to be declared in a file that can be shared in the project’s source control repository. Everything needed to setup the environment is declared in a file called Vagrantfile and when a developer wishes to set up his he just executes vagrant up.

A few minutes later, a fully operational environment is running inside a virtual machine which in turn runs on his desktop environment. In a MacOS X or Linux environment, the developer usually edits files by running his editor on the host system but builds, deploys and tests the system in the virtual machine. Moreover, Vagrant allows for provisioning scripts declared in the Vagrantfile but it also allows integrating with a Puppet or Chef infrastructure. Which means that the development virtual machine can be provisioned exactly like the test and production environments.

It also allows for a nice DevOps pipeline by permitting each developer to experiment with an infrastructure change on his own virtual machine. When satisfied she can then share that change through source control. Her fellow developers will benefit from the change by refreshing their sources and executing a simple vagrant provision command. The change can then be merged, again through source control, in the test and production environments. Developers and operation people can experiment in their environment without fear and at a very low cost. Breaking their environment would only mean that they have to revert their changes through source control and execute a vagrant destroy; vagrant up command (with the additional benefit that they get to grab a cup of coffee while this is running).

However, after a few experimentation with Vagrant on Windows, we quickly found out that the developer experience was not satisfactory. The performance was very degraded, editing on Windows while building on Linux proved to be quite challenging. On the other hand, the JEE technology with thousands of Java, XML and ZIP files was posing a greater challenge to virtualized environment than the more common Rails, Node or Python environments.

As we already planned on using Puppet for test and production environment deployment, we settled on deploying each developer’s desktop environment with the same tool. This allowed us to almost completely automate desktop environment setup. It had a few drawbacks, though. Mainly, for those project members that worked on their laptop and had previously installed tooling, the setup process might interfere with those tools or fail altogether because they were set up in an unexpected way. We also had the common issues that were found out only in the test environment because developers tested on their own Windows environment.

Step 2: Deploy on Windows in a Linux shop

The second step was a bit different. This time, we were both developing and deploying on Windows. The project was only a few months old; the deployment process was manual, and configuration management was starting to become an issue. Most of the rest of the company was deploying applications on Linux, and the infrastructure team had already introduced Puppet to manage those hosts. We ourselves used Puppet to deploy our hosts to build on existing experience and to prepare for an eventual migration from Windows to Linux.

All environments were managed by the same system which allowed developers to have local setups almost identical to test and production environments. Deployments and configuration changes were propagated along the pipeline by successively merging from development to test and from test to production. Configuration data was stored in HieraDB a part of Puppet that provides a hierarchical configuration database.

This particular software, we were developing, required complicated configuration management, and we rapidly reaped the benefits of introducing automation. Especially when deployments started multiplying.
However, it revealed much more vividly than in step 1 that Puppet, or any tool that grew up in the Linux world, has an impedance mismatch when deployed on Windows. Basically, Unices are based on files and executables whereas Windows is much more complex. The registry, the right management system, drivers, host processes, services, etc. All those concepts are not hugely different from Linux equivalents. But nonetheless, different enough to make things complicated with tools like Puppet, Chef, Ansible, Salt, etc. Nothing is completely impossible, but you have to work harder to achieve the same result on Windows than in their native OSes.

Especially, when you cannot handle the task with pre-existing resources and need to add some new behavior to the configuration management tool. This can be achieved either by extending them with Ruby code for Puppet and Chef or Python code for Ansible and Salt. But even general purpose languages like Ruby and Python that are ported to Windows show some of that impedance mismatch. Additionally, most of the time, they are not known by existing IT professional. We chose to use Bash through the MSysGit toolbox because both developers and IT professionals were familiar with it. However, that still was not satisfactory. Even if that version of Bash is very well integrated with Windows, it still does not work seamlessly. Very simple tasks like creating a properly configured Windows service required juggling with different concepts and commands.

Note: in the meantime, Microsoft has released a beta version of Bash on Windows which is another version of Bash running on Windows. We did not use it as much as the MSysGit version, but it seems to exhibit the same inherent difficulties.

Step 3: What the heck, let’s learn PowerShell

The latest step of our journey occurred in a full Microsoft environment. Development and deployment happened almost exclusively on Windows. Modern configuration management like Puppet or Chef was not yet introduced. However, several efforts were made to introduce automation, but mostly of the mechanization sort (see below). Repetitive tasks have been accumulated together into scripts, in some cases, those scripts are called through workflow like graphical interfaces, but mostly, the control and knowledge remained with the IT professional.

This time around, people already were familiar with PowerShell, the scripting language introduced by Microsoft in 2006. The approach we took was to expand on that existing body of knowledge with extensive training and systematic exploration of PowerShell solutions to our issues before even considering more advanced but also more alien solutions. We found that PowerShell provided most of the required building blocks for a full deployment and configuration management stack.

PowerShell

PowerShell is a scripting language introduced by Microsoft in 2006. But it was already laid out in the Monad Manifesto by Jeffrey Snover in 2002. As explained in the manifesto, PowerShell takes a new approach to old issues. It is a shell with a pipeline. However, PowerShell passes objects down the pipeline, not text or binary data like its Unix relatives. This saves quite a bit of work when composing simpler commands together.

PowerShell also leverages the .Net framework which makes it as powerful as any general purpose programming language. Access to the .Net framework means that any API available to .Net programmers will also be accessible through PowerShell. But it does not stop there. It also provides jobs, remoting, workflows, package managment and many other features. Among those, two are of particular interest when building an automation solution: Desired State Configuration and Just Enough Administration.

Desired State Configuration

DSC builds on top of PowerShell to implement configuration management building blocks called resources. Those resources, are very similar to Puppet resources and similarly allow IT professionals to declaratively express what the state of the system should be. The actual imperative implementation that changes the system to the desired state is implemented by a PowerShell module.

DSC also provides a declarative domain specific language to express a system’s configuration and an agent that can apply that configuration to a specific host. The agent is an integral part of the Windows Management Framework which means that it will be deployed everywhere PowerShell is.

Transforming traditional imperative scripts into DSC resources allows us to achieve idempotence. Which in turn allows us to handle any configuration change like a simple change in the declared configuration. DSC will then ensure the system ends in the required state from the previous state. Even, a manual change, of the system would be caught by DSC as a configuration drift and could be automatically corrected, depending on the agent’s configuration.

DSC can do much more. However, it is not as mature a solution as Puppet or Chef. It lacks the ecosystem around it: a configuration database, reporting tools, easy but powerful configuration composition and reuse, etc. Therefore, our current approach is to associate both tools. The impedance mismatch mentioned earlier can be solved by letting Puppet or Chef use DSC resources to effect configuration changes on Windows, creating a win/win situation. Benefit from all the power behind the Puppet and Chef ecosystems, while still leveraging PowerShell, which is native to Windows and already known to IT professionals.

Our current preference is to use Chef. Chef is a bit less mature than Puppet, but it seems that Chef and Microsoft actively work together to integrate both solutions.

Just Enough Administration

JEA is a tool that helps reduce administrative rights dissemination. Lots of operations require administrative or at least somehow elevated rights to be performed. That usually means that if you have to perform one of those operations (even if just is a read-only operation), you will get administrative rights. Or some ad hoc solution would be put in place to somehow allow you to do what you needed to. This can lead to pretty complex and fragile solutions.

From my Linux background point of view, JEA is like sudo on steroids. It handles both privilege elevation, RBAC, remoting and integrates nicely with PowerShell scripting. It can easily be deployed via DSC which helps to ensure limited and uniform access rights throughout the whole system.

Perspective

That latest step is still quite new and, for some parts, a work in progress. We have solved all the issues encountered in previous experiences and even a few new ones, which leads us to think we are on the right track. Moreover, it seems the vision we have created aligns nicely with the “new” Microsoft. For instance, the recent open sourcing of .Net and PowerShell, while at the same time porting it to Linux and MacOSX, opens new avenues for automating our few existing Linux systems.

Lessons learned

The following lessons were learned the hard way. They are a bit opinionated. But, all are rooted in real life situations where not following them meant failure…

Start small.

In my experience, trying to automate the whole system at once ends badly. At best, only those parts of the system that were easy end up automated, leaving huge gaps where manual intervention is still needed. At worst, the effort fails completely.

Start small by considering only a single part of the system. However, ensure that that part is entirely automated. Do not leave some manual steps in-between automated parts. The automation should encompass all use cases, even those considered exceptions. If it looks like they are too different to be automated, it usually means that the automation system is too rigid, too specialized to some local or current way of doing things. Not being able to handle today’s exceptions is a good indicator that the system we are creating will probably not be able to handle future either. Also, partial coverage means that we cannot have full confidence in our ability to reconstruct the system. Be it because we are facing a major disaster or, just because we need to experiment with a copy of the actual system.

This makes for slower progress but allows you to get a solid foothold on the automated ground. Expanding from that solid ground will be much easier in the future.

“Mechanization” is not automation…

In my experience, in environments where the concept of automation is new, it is often confused with mechanization. It is what might be called the reduce 10 steps to 1 syndrome.

Wikipedia defines Mechanization as:

Mechanization is the process of changing from working largely or exclusively by hand or with animals to doing that work with machinery.

Too often, automation is confused with mechanization. Allowing developers to submit an SQL schema migration through a form is a form of automation. The developer does not need to open a SQL tool, enter the proper credentials and execute the SQL file by hand. However, it is not automation yet. The process is still almost exclusively under the control of a human being (animal labor is now mostly eliminated from our IT operations, although we still occasionally see some developers talking to ducks). Wikipedia defines Automation as:

Automation is the use of various control systems for operating equipment […] with minimal or reduced human intervention.

To fully automate our SQL database schema migrations, human intervention should be entirely eliminated. Which means that the developer should only specify which version of the database schema is required and automation will take care of bringing the database to the proper state. Tools like Liquibase or Flyway can help a lot with that. To the point that humans do not even need to ask for a database schema version change. The application can, upon starting, check that the database is in the proper state and, if not, apply the relevant migrations.

While introducing automation instead of mechanization is a bit harder it has tremendous advantages down the road. To quote Wikipedia one last time:

The biggest benefit of automation is that it saves labor; however, it is also used to save energy and materials and to improve quality, accuracy, and precision.

Automation completely eliminates human error by eliminating human decision from the process. Which in turn improves quality and repeatability.

Keep away from shiny tools

With the introduction of Infrastructure as Code IT professionals became de facto developers. That means their activity has changed from simply deploying and operating software to actually building the software that deploys that software -even when that software is some piece of infrastructure.

That poses a challenge to any IT professional that did not start his career as a developer. However, in the Linux world, any IT professional has been exposed to a variety of scripting languages which performed a lot of tasks from the most mundane to some quite sophisticated (for instance Gentoo’s build system, ebuild). The Microsoft world, however, has long relied on graphical user interfaces and manual operations. It is thus natural to turn to graphical tools that promise to simplify infrastructure as code by allowing building complex automation through graphical manipulation. I have yet to find a graphical tool that fulfills that promise. During the last years, I found quite a few instances where they made things way worse.

Coding is coding. To this day, most of the coding occurs by writing code. Yes, there are a few specialized tasks that are better handled by graphical tools, but general coding tasks usually involve editing some form of text. So, even if you can leverage graphical tools for some of your automation tasks, in the end, a general purpose scripting language will always be necessary.

The language of choice, on Microsoft Windows, is PowerShell. It is still a Shell with its roots firmly planted in the Unix world but is also very well integrated with the Windows system and Windows applications. All of which makes your life so much easier. If you only learned PowerShell by doing the same stuff you did with .BAT files and sprinkling a bit of Stack Overflow, try to get some formal training. It will be time and money well spent that will pay for itself time and time again.

Involve IT professionals

Automation should be built by the people who are the most intimate with the system being automated. If the automation solution is built by outside people, it often lacks the proper adoption that would permit it to grow. In some extreme cases, the knowledge of what the automation actually does is lost in the system. Developers move on to building other systems, and IT professionals lack the expertise to actually dig into the system when needed.

Of course, in most occurrences, IT professionals are not yet developers and lack the set of skills required to build the automation tools. In which case, bringing in external resources to help mentor them in those new fields will speed things up and prevent fundamental mistakes. However, at the end of the process, IT professionals should be in charge and complete control of their toolchain.

One way to flatten their learning curve while still leaving them in charge is to help them set up a software factory and software tooling very early in the process. If PowerShell module scaffolds can be generated, tests automatically executed, modules packaged and deployed automatically to the repository, etc. they can concentrate on what really matters: the automation system they are building.

The release pipeline model

Earlier this year Microsoft published a white paper about the Release Pipeline Model which neatly sums up everything we mentioned here (and more). I strongly encourage anyone attempting an automated deployment and configuration management effort in the Microsoft world to read it. Most of what you would need to start, but is not mentioned in this post, can be found in it.

Looking forward

It is probably too soon to gain enough perspective on how automated configuration management will evolve in the Microsoft world. On the other hand, all pieces seem to fall neatly in place and work well together.

Day 2 - DBAs, a priesthood no more

Written by: Silvia Botros (@dbsmasher)
Edited by: Shaun Mouton (@sdmouton)

Hermione casting a spell.
Illustration by Frida Lundqvist

Companies have had and needed Database Administrators for years. Data is one of a business’s most important assets. That means many businesses, once they grow to the point where they must be able to rapidly scale, need someone to make sure that asset is well managed, performant for the product needs, and available to restore in case of disasters.

In a traditional sense, the job of the DBA means she is the only person with access to the servers that host the data, the person to go to create new database cluster for new features, the person to design new schemas, and the only person to contact when anything database related breaks in a production environment.

Because DBAs traditionally have such unique roles their time is at a premium, and it becomes harder to think big picture when day to day tasks overwhelm. It is typical to resort to brittle tools like bash for all sorts of operational tasks in DBA land. Need a new DB setup from a clean OS install? Take, validate, or restore backups? Rotate partitions or stale data? When your most commonly used tool is bash scripting, everything looks like a nail. I am sure many readers are preparing tweets to tell me how powerful bash is, but please hold your commentary until after evaluating my reasoning.

Does all this sound like your job description as a DBA? Does the job description talk in details about upgrading servers, creating and testing backups, and monitoring? Most typical DBA job postings will make sure to say that you have to configure and setup ‘multiple’ database servers (because the expectation is that DBAs hand craft them), and automate database management tasks with (hand crafted) scripts.

Is that really a scalable approach for what is often a team of one in a growing, fast paced organization?

I am here to argue that your job is not to perform and manage backups, create and manage databases, or optimize queries. You will do all these things in the span of your job but the primary goal is to make your business’s data accessible and scalable. This is not just for the business to run the current product but also to build new features and provide value to customers.

Why

You may want to ask, why would I do any of this? There is an argument for continuing the execute the DBA role traditionally: job security, right?

Many tech organizations nowadays do one or more of the following:

• They are formed of many smaller teams
• They provide feature by creating many micro-services in place of one or a few larger services
• They adopt agile methodologies to speed the delivery of features
• They combine operations and engineering under one leadership
• They embed operations engineers with developers as early as possible in the design process

A DBA silo within operations means the operations team is less empowered to help debug production issues in its own stack, is sometimes unable to respond and fix issues without assistance, and frankly less credible at demanding closer and earlier collaborations with the engineering teams if they aren’t practicing what they preach inside Tech Ops.

So what can be done to bust that silo and make it easier for other folks to debug, help scale the database layer, and empower engineers to design services that can scale? Most up-and coming shops have at most one in-house DBA. Can the one DBA be ‘present’ in all design meetings, approve every schema change, and be on call for a sprawling, ever growing database footprint?

DBAs can no longer be gate keepers or magicians. A DBA can and should be a source of knowledge and expertise for engineers in an organization. She should help the delivery teams not just deliver features but to deliver products that scale and empower them to not fear the database. But how can a DBA achieve that while doing the daily job of managing the data layer? There are a number of ways you, the DBA, can set yourself up for excellence.

Configuration management

This is a very important one. DBAs tend to prefer old school tools like bash for database setup. I alluded to this earlier and I have nothing against using bash itself. I use it a lot, actually. But it is not the right tool for cluster setup. Especially if the rest of ops is NOT using Bash to manage the rest of the architecture. It’s true that operations engineers know Bash too, but if they are managing the rest of the infrastructure with a tool like Chef or Puppet and the databases are managed mostly by hand crafted scripts written by the DBA, you are imposing an obstruction for them to provide help when an urgent change is needed. Moreover, it becomes harder to help engineering teams to self-serve and own the creation of the new clusters they need for new feature foo. You become the ‘blocker’ for completing work. Getting familiar with the configuration management at your company is also a two way benefit. As you get familiar with how the infrastructure is managed, you get to know the team’s standards, get more familiar with the stack, and are able to collaborate on changes that ultimately affect the product scale. A DBA who is tuned into the engineering organization’s product and infrastructure as a whole is invaluable.

Runbooks

This is technically a subset of the documentation you have to write (you document things, right?!) but in my experience has proven far more useful that I feel it has to be pointed out separately. When I say runbooks I am specifically saying a document written for an audience that is NOT a DBA. There are a lot of production DB issues we may encounter as DBAs that are simple for us to debug and resolve. We tend to underestimate that muscle memory and we fall in the pattern of ‘just send me the page’ and we ‘take care of things’.

If your operations team is like mine where you are the only DBA, it probably means someone else on the team is the first line of defense when a DB related event pages. Some simple documentation on how to do initial debugging, data collection, can go a long way in making the rest of the operations team comfortable with the database layer and more familiar with how we monitor it and debug it. Even if that event still results into paging the DBA, slowly but surely, the runbook becomes a place for everyone to add acquired knowledge.

Additionally, I add a link to the related runbook section (use anchors!) to the page descriptions that go to the pager. This is incredibly helpful for someone being paged by a database host at 3 AM to find a place to start. These things may seem small but in my experience they have gone a long way breaking mental barriers for my operations team working on database layer when necessary.

As a personal preference, I write these as markdown docs inside my Chef cookbook repositories. This falls seamlessly into a pull request, review and merge pattern, and it becomes an integral part of the databases’ cookbooks pattern. As engineering teams start creating their own, the runbooks become a familiar template as new database clusters spring out all over the place.

Visibility

We like our terminal screens. We love them. The most popular tools in MySQL land are still terminal tools that live directly on the db hosts and that need prior knowledge of them and how to use them. I am talking about things like innotop and the MySQL shell. These are fine and still helpful but they are created for DBAs. If you do not want to be the gatekeeper to questions like “is there replication lag right now”, you need to have better tools to make any cluster health, now and historically, available and easy to digest for all team members. I have a few examples in this arena:

Orchestrator

We use read replicas to spread that load away from the primary, which means once lag hits a certain threshold, it becomes a customer support event. It is important to make it easier for anyone in the company to know at any given time whether any cluster is experiencing lag, what servers in that cluster are, and whether any of the hosts has gone down. Orchestrator is a great tool in that regard in that it makes visualizing clusters and their health a browser window away.

Grafana/Graphite

Metrics for the DB layer need to live in the same place metrics for the rest of the infrastructure are. It is important for the team to be able to juxtapose these metrics side by side. And it is important to have an easy way to see historical metrics for any DB cluster. While you may have a personal preference for cacti or munin, or artisanal templates that you have written over the years, if the metrics you use to investigate issues are not in the same place as the rest of the infrastructure metrics it sets up a barrier for other busy engineers, and they’ll be less inclined to use your tooling over that which is in use elsewhere. Graphite is in wide use for ingesting metrics in modern infrastructure teams, and Grafana is a widely used dashboarding front-end for metrics and analytics.

Query performance

We use VividCortex to track our queries on critical clusters and while this article isn’t meant to be an advertisement for a paid service, I will say that you need to make the ability to inspect the effect of deploys and code changes on running queries and query performance something that doesn’t need special access to logs and manually crunching them. If VividCortex isn’t a possibility (although, seriously, they are awesome!), there are other products and open source tools that can capture even just the slow log and put it in an easy to read web page for non DBAs to inspect and see the effect of their code. The important point here is that if you provide the means to see the data, engineers will use that data and do their best to keep things efficient. But it is part of your job to make that access available and not a special DBA trick.

Fight the pager fatigue

A lot of organizations do not have scaling the database layer as a very early imperative in their stack design, and they shouldn’t. In the early days of a company, you shouldn’t worry about how you will throttle API calls if no one is using the API yet. But it’s appropriate to consider a few years later, when the product has gained traction, and that API call that was hitting a table of a few thousand rows by a handful of customers is now a multi million rows table, and a couple of customers have built cron jobs that flood that API every morning at 6 AM your timezone.

It takes a lot of work to change the application layer of any product to protect the infrastructure and in the interim, allowing spurious database activity to cause pager fatigue is a big danger to both you and the rest of the operations organization. Get familiar with tools like pt-kill that can be used in a cinch to keep a database host from having major downtime due to unplanned volume. Make the use of that tool known and communicate the action and its effect to the stakeholder engineering team but it is unhealthy to try and absorb the pain from something you directly cannot change and it is ultimately not going to be beneficial to helping the engineering teams’ learn how to deal with growing pains.

There are a lot of ways a DBA’s work is unique to her role in comparison to the rest of the operations team but that doesn’t mean it has to be a magical priesthood no one can approach. These steps go a long way in making your work transparent but most importantly is approaching your work not as a gatekeeper to a golden garden of database host but as a subject matter expert who can provide advice and help grow the engineers you work with and provide more value to the business than backups and query tuning (but those are fun too!).

Special thanks to the wonderful operations team at Sendgrid who continue to teach me many things, and to Charity Majors for coining the title of this post.

Day 15 - Fear and Loathing in Systems Administration

Written by H. “Waldo” Grunenwald (@gwaldo)
Edited by Shaun Mouton (@sdmouton)

“DevOps Doesn’t Work”

The number of times that I’ve heard this is amazing. The best thing about this phrase is that the people who say it are often completely right, even if for very wrong reasons.

Who Says This?

Well, let’s talk about the people who most commonly have this reaction: SysAdmins. I’m going to use the term “SysAdmins” as a shorthand for a broad group. The people in this group have widely varying titles, but it is most commonly “Systems”, “Network”, or “Operations” follwed by “Administrator”, “Engineer”, “Technician”, or “Analyst”.

In some companies, these folks have the best computers in the place. In others, they have to live with the worst. Their workspace probably isn’t very nice, and almost certainly has no natural light. If there is a pager rotation, they are almost certainly on it. If there isn’t a rotation, they’re basically on-call all of the time.

During the course of a normal day they might have to switch contexts between disaster planning, calculating power and HVAC needs for a new datacenter, scrambling to complete an outage-driven SAN migration, rushing to address urgent requests to help people with their email, to troubleshoot a printing problem, or suss out why someone can’t get to their electric company’s bill pay website. They may be the sole person with database expertise in the company, or they may work on a team of dozens.

The work is largely invisible except when something fails, in which case it’s highly visible and widely impacting.

These are typically cynical people, because there are only so many times that you can’t make the team/department/company party for ostensibly “celebrating our successes” because something’s broken, and you’re left to clean up after the “success”. There are only so many times that one sees a new project announced and begins to hire more people. When asked who’s going to support the new project, the response is a blank look and “you are”. The “…of course.” may not be vocalized, but it’s probably there. When asked how many people they get to hire to help with the workload, the response is a combination of “sorry, but there wasn’t anything left in the budget”, “it won’t be that much more work”, or a variation of the “team player / good soldier” speech. There are only so many times one can take getting your requests for training or conference budget rejected out of hand, and have your requests for training or conference budget laughed out of the room.

They probably have basic working knowledge of a half-dozen programming languages, but most likely they often think in Shell. They probably know at least three ways of testing if a port is open, and probably have a soft spot in their heart for a couple of shell commands.

They may have seen or participated in a DevOps initiative that consisted of a team or position rename, or helpfully suggested that they install some Config Management and Monitoring software so that “we can DevOps now…” or “so we can do Agile”. When they hear “DevOps” or “Agile”, what they are hearing is is Let’s take the same people who can’t handle a planned release schedule or make whatever effort that they need to squeak by the Change Board and Release Management requirements, and give them unfettered access to Production. Clearly, I’m not paged often enough.

So what is one to do? How is one to maintain their sanity in the face of increasing job scope, increasing demand for access and velocity, and little hope for an effective new-hire count? Not to mention continuing to juggle the existing volume of requests, and continuing to grease the existing gears to keep the machine running.

Get Help

Please note that I’m not saying “just”. There’s nothing just about this situation; there is nothing simple about any of this, and Justice hasn’t been seen in a long time in an environment where this is the norm. Most of these changes are difficult. They will take work, and will require convincing other teams to join in your cause.

Admitting you have a Problem

The problem (probably) isn’t technical. It’s almost entirely social.

Because SysAdmins are typically responsible for the environment, the easiest way to assure that the state is stable is to lock everyone else from it. While this helped with the goal of “keeping out unexpected changes”, it had a number of side effects.

First, a kind of learned helplessness has set in. Your customers and teammates became so used to being “hands-off”, that they don’t have the wherewithal to meet reasonable expectations. Since they’re uncomfortable making any changes, all changes must be made by the SysAdmins. This leads to your time being taken by having to perform lots of low-value tasks.

Some teams settle on the pattern of “hands off Production, but you have access to Staging”, but this is fraught with peril. The most common problem that stems from this is “Configuration Drift.” Config Drift is when you have different settings in one environment (or server) than the others. When the cost to discover what Production looks like is high, it’s more likely that people will either use defaults, make assumptions, or use the same configs that they use in their IDEs. “Works on my machine”, indeed.

This is a problem well-solved by Configuration Management tools, but you still need to be willing to trust your peers and give them access. If you want to be part of the process of validating changes, you could put in place the structures that allow a pull-request and code-review workflow, something that your Software Engineering peers should be very accustomed to! Granting access to see the existing configs and the ability to propose changes also shares responsibility for your team’s environments and contributes to feelings of ownership. Denying colleagues the ability to effect necessary configuration changes contributes to the root problems of configuration drift and learned helplessness.

Stop Feeding the Machine

Your value is not in doing the work, but rather being able to make the decision to do the work.
I’ll be the first to say that “Automating ALL THE THINGS” is a flawed goal. At work, it’s usually said in the context of a Project, rather than part of a philosophy of Continual Improvement (Think Toyota Kata). You shouldn’t have to engage in an “Automation Project” to improve your environment. Build into your schedule time to solve one problem. Pick something that is rough, manual, and repeatable. Remove a small piece of friction. Move on to the next one. Hint: Logging into a server to make a configuration change should be a cue to implement configuration management!

While I agree that everything being automated, not everything should be automatic. Decision-making is complex, and attempting to codify all of the possible decision-making points is a fantastic way to make yourself insane. Not to mention that documenting your decision-making processes may be an unwanted look inside your brain. Caveat Implementor. (Or perhaps that’s just me…)

All of the units of work should be automated. But the decision to run the now-automated tasks can be left to a human. When you find that there is a correlation between steps, those pieces should be wrapped together. Automation isn’t a project into itself. It should be iterative. Pick something that’s painful. Make it a little smoother. Repeat. Ideally, you have time blocked out for Continuous Improvement. If not, create a meeting, or create a weekly project to do so. Review the issues that you’ve experienced lately, and pick something to make better. It might be worth making into a project, but it won’t be an ALL-THE-THINGS project. Create a scope of effort. Take the time to plan goals and requirements.

Whatever you don’t automate must be documented. Beyond the typical benefits of documentation, it also serves as “Functional Requirements” for someone else to pick up when they can help you with providing a solution. Try to recognize whether documenting or automating takes longer. Perhaps this piece of documentation will bet better served by “Executable Documentation” (i.e. code).

Clarify Your Role

You should attempt to pick apart the parts of your work, and attempt to describe them. One way to make this a fun exercise is to use other job titles to describe the work.

Are you an “Internet Plumber”? How much of your job could be described as “Spelunking” into the deep dark caverns of Legacy systems?

If you want, you could ascribe Superhero names to these parts of your work. The added bonus is that it not only describes a role, but also a demeanor associated with them. When ‘bad code’ makes it to Production, do you go “Wolverine” on that dev team?

Could you describe part of your role as “Production Customs Official”? Are you the gateway to Production? If so, are you actually equipped to do that? Here’s a quick test: When you say “no, that can’t go”, do you get overridden?

More importantly, is this what you want to do?

Prepwork

You will need to prepare for this. Most SysAdmin teams do not have a healthy relationship with the rest of the business. You will need to initiate the healing.

Take someone to lunch. Preferably someone who you don’t know well. Ask questions, and listen to the answers. It is not time to defend yourself or your team. It’s time to find out what the business needs from someone else’s perspective. Ask what they think that your team’s role is in toward achieving that success. Ask what they think your team does well, and where there are gaps between what you have now and excellence.

Speak their language

You probably recognize their words, but you need to go out of your way to speak them. To communicate your message, you will need meet them on their turf. This may seem terribly unfair - “Why can’t they meet me on my terms?!” - but I’m guessing that has not been working out well for you so far.

Not only do you need to use their language, but you need to communicate over their medium. And identifying who they are is step one in learning to speak it. It’s probably not IRC, and only writing it in email is a good way for it to be ignored.

If you’re speaking to management, be prepared to write a presentation. Executives especially like to see a slide-deck. It doesn’t have to be slick. It probably shouldn’t have sounds or much in the way of transitions, but a presentation can help to lay the groundwork for a conversation.

Discuss Scope, Staffing, and Priorities

Now that you have described your role, we also need to describe everything that you support.
What Products do you support? It’s entirely possible (likely, even) that the people and teams that you support don’t actually know what you’re responsible for. It could be argued that most of them shouldn’t need to know. But if you have been saying “no” to protect yourself, it’s a sign that you are significantly overextended. You need to have a real discussion with your leadership about your role, scope, and staffing.

In order to have this discussion, you need to prepare. You need to come up with a fairly comprehensive list of the products and teams that you support. This is a list of every team, and their products, the components and tasks that belong to you for each. Don’t forget all of the components that “nobody owns” but somehow people come to you to fix or implement (CI, SCM, Ticketing, Project, and Wiki tools seem to be common examples). Are you also responsible for Directory Services? Virtualization platform? Mail/Chat/Phones? Workstation Purchasing and provisioning? Printers? Do you manage the Storage, Networking, etc? Don’t be afraid of getting into details. It can help to provide clearly written potential impacts the company if some of these “hidden” services stop working? Your leadership might not know what LDAP or Directory Services are, but they’ll understand if nobody can log into their machines, they can’t pull information to build reports, and by-the-way nobody can deploy code because it relies on validating credentials…

What is most important to the company? What do you need to succeed? How much more staff do you need? What tooling or equipment would help you work more efficiently? Does code deploy even when it fails testing? How many outages have arisen due to this happening?

Demonstrate Cost and Value and Revisit Priorities

In order to have meaningful discussions with people in your company who aren’t necessarily technical, you need to be able to relate to a language that they speak. Regardless of team duties, the lingua franca of most teams is money. As Engineers, most of us prefer to think in terms of the tech itself, but in order to describe an impact, a unit of monetary value is a proxy for impact that most non-technical people can understand, even if they don’t grasp the details.

It is a helpful (if difficult and uncomfortable) habit to get into, but I encourage you to consider the components of cost that goes into every incident or task.

What is the cost of a main-site outage? How much revenue does this feature bring in? Why are you spending so much on infrastructure and effort to make that component Highly-Available? Why does it matter that you do that piece of maintenance? Show the negative value of doing things they way they are (Opportunity Cost), versus investing time to improve the automation around it. Describe how doing this maintenance work reduces your context switching, unplanned outages, and lost reputation of your company. Describe the benefit in increased visibility to the business, and Agency to be gained by your peers on other teams.

Why put in place these tools to let product teams self-serve? Describe that the features that the company’s teams spend so much time and effort (read: “money”) creating means nothing if those features aren’t available for customers to use. That having those features not available costs money in terms of feature billing, and reputation cost. If they claim that they’re doing Agile, but can’t do Continuous Delivery, they’re not really Agile, and the whole point of that framework is to improve delivery of value to the customer and the business!

Further, show how systems relate. It doesn’t have to be terribly detailed. Describe that the features that the customers use are reliant on x, y, and z components of infrastructure. Draw the lines from LDAP to storage to your CI tool to testing code to artifacts delivered to Production. Then show some of the other systems that have similar dependencies.

Once the picture emerges showing how everything is reliant on unexciting things like LDAP, your Storage cluster, and that janky collection of angry shell and perl scripts that keep everything working, realization will begin to dawn.

Congratulations, you’ve just effectively communicated Value.

Align Responsibility with Authority

Are you held responsible for apps written by other people? Who gets paged when “the app” goes down? How does that make sense?

Get Devs on-call for their apps. SysAdmins should be escalated to. Devs can triage and troubleshoot their own apps more readily than you can. They get to call in the cavalry when they get stuck. They don’t need to know everything about the systems, and they don’t need to resolve everything. When a fault occurs and they need help, they stay on the call, pairing with you as you diagnose, troubleshoot, and resolve. That way, they don’t need to escalate to you for that thing the next time it occurs, and can collaborate on automating a permanent fix.

When teams aren’t responsible for their products - When they aren’t paged when it fails - they are numb to the pain that they inflict. They’re not trying to cause pain; they just don’t feel it. It’s especially easy to argue this for teams that proclaim that they use Agile development methods: If they claim to want “continuous feedback”, there is nothing more visceral for providing feedback than the feeling of being awoken by a pager in the middle of the night. When the inevitable exclaimation comes that “we can’t interrupt our developers”, ask if it makes sense to interrupt someone else.
Even being aware of the pain (say, hearing how many times you were paged last night) can elicit sympathy, but that’s a far cry from the experience of being paged yourself.

Further, this is what that list of responsibilities is for. Asking each team to take responsilbity for their own products, you will still likely have a hefty list of services that you provide that you are on-call for. As these set in, point out the staffing numbers. This may be a matter of the places that I have worked, but I have never seen a Developer-to-SysAdmin ratio of less than 5-1. In most places it is much higher. By adding these teams to pager rotations, they drastically reduce the load on you. By not adding them to pager rotations, they are complicit in your burnout.

Stop saying “No”

SysAdmins have a reputation for saying “No”. The people who are asking are probably not trying to make your life worse; They’re probably just trying to get their work done. They might not know what their “simple request” involves, and that it probably isn’t necessary.

But by not having Responsibility aligned with Authority, you may have been stuck with the pain of other people’s wishes. You know that fulfilling their request will cause you pain, so understandably, you say “no”. What often happens next is that they escalate until they hit someone sufficiently important enough to override you.

This is the basis for why SysAdmins feel steamrolled by everyone else, and everyone else feels held hostage by SysAdmins.

But all hope is not lost.

Stop saying “No”.

“Yes, but …” is a very powerful thing.

“Yes, but …” can be used to get you help.

“Yes, I can set that up for you, but we don’t have capacity to run it for you.” What happened there? You agreed that the request is reasonable. You set expectations of the level of support that you can give. You left the requestor with several options to continue the conversation.

• They might have hiring reqs that they can’t fill. You can negotiate for some of them to go to your team, as you’re clearly understaffed.
• Some of their engineers may join your team as a lateral move. They’ll need mentorship and training, but this kind of cross-training is invaluable. It’s a force multiplier. It also sets precedent.
• They might take the responsibility for the Thing. They run it. They get paged for it. Of course you will probably have to be an escalation point to assist when it fails, but it’s their product. This again sets precedent.
  

Delegate

Most SysAdmins are stuck doing tasks that provide very little value because they restrict access to their peers. To my mind, there is one perfect example: “Playing Telephone”.

When I say “Playing Telephone”, I’m talking about the situation where someone (let’s say a Developer) wants logs from the application, but they don’t have access to get them. They request the logs from you. You fetch the log requested and provide it to them. “No, not that log, this log…” You fetch. “Hmm, I’m not seeing what I’m looking for, could you check in here for something that says something like this …?” And so on, and so on…

I don’t know what you’re hoping to prevent by restricting access, but if this scenario ever happens, you should know that you’re providing Negative Value. Again, let’s try to remember that your peers are not out to get you, and can probably be trusted to be reasonable humans if you meet them mid-way.

With that framework in mind, it’s time that you demonstrate some trust, and Delegate to them. Give them access. Your value is not in the logon credentials that you have, otherwise you’re just a poorly-implemented “Terminal-as-a-Service”.

Even better than giving access, is giving Tooling. Logging into a server should be an antipattern for most work! You need some better tooling. So, with the example of logging, let’s talk tooling.

Logging

First, logging into boxes to get logs is just dumb. Sure, you could wrap a tail command in a Rundeck job, but let’s Centralize those logs while we’re at it.

SysLog is better than nothing, but not by much. Shipping logs is easy, but consuming them as something useful is not. Batteries not included.

If your company wants to spend the money on Splunk, then encourage that. Splunk is a fantastic suite of tools, but I might wave you away from it if you’re not going to use it for everything. It’s going to be expensive, and if you’re not going to spend enough to use it for everything, there will be confusion as to what’s in there, and what’s stored elsewhere.

ELK (ElasticSearch + Logstash + Kibana, sometimes mistakenly simplified to “Logstash”), or “Cloudy Elk” / “ELK-as-a-Service” is a good middle-ground. ELK is Free (as-in-beer), and very featureful.

Take your Centralized logging of choice, and provide your customers with the url to the web interface. Send them links to the “How to use” docs, and get out of their way!

Terminal-as-a-Service

If someone asks you to “run this command for me”, you need to put a button on it.

You don’t need to RUN-ALL-THE-THINGS!

Rundeck is a fantastic tool to “Put a button on it”. Other people use their CI tools (like Jenkins or Bamboo) for this. My friend Jeremy Price gave an Ignite Talk at DevOpsDays NYC 2015 that describes this.

Personally I like Rundeck, because it’s pretty easy to make HA, tie it into LDAP for credentials, manage permissions, and by shipping it’s logs (see what I did there?), you get Auditing of who ran what and when!

If you have some data that Must be restricted, try to isolate those cases from the rest of your environment. You shouldn’t have to restrict Everything just because Something does need isolation.

Deploying Code. Yes, to Production

Why would you want to have to deploy other people’s code?! Do you really provide any value in that activity? If the deployment doesn’t go well, you’re launching another game of “Telephone”.
What if you make it easy for them to do it? Empower them with trust and tooling, making it easy to do the right thing! Give them tooling to see that the deploy succeeded! Logs are a start, but Metrics Dashboards that show changes in performance conditions and error rates will make it plain to see if a deployment was successful!

This Freedom doesn’t come free. Providing tooling doesn’t absolve the development teams of the need to communicate; in fact, it’s likely that they’ll have to communicate more. They will need to be watching those dashboards and logs to see for themselves the success of every deploy. They will also be more readily on-hand to help triage the inevitable instances when it doesn’t go swimmingly.

Us

I say “They” in this article a lot. And that is because, by default, most organizations that I have been a part of or heard stories of have had a strong component of “Us-Versus-Them.” It’s only natural for there to be an “Us” and a “Them”, but thinking in those terms should be a very short-term use of the language. Strive for the goal of a “We” in your interactions at work, and reinforce that language wherever possible. While it may not be My job to do “foo”, it is Our job to ensure the team and company is successful.

While that may sounds like some happy-go-lucky, tree-hugging, pop-psychological nonsense (and it is…:), the goal here is to get you, the beleaguered SysAdmin the help that you need, in order to improve the capabilites of the business.

Coda

There is so much more to this topic, particularly the shift away from a Systems team supporting a bunch of Project teams to a series of largely self-sustaining Product teams, but that will have to wait for another day.

The psychological damage done to SysAdmins by their peers can make us bitter and cynical. I encourage my people to try to see that “They” aren’t trying to make life difficult for you, but it’s very likely that Authority and Responsibility are misaligned. I likewise encourage my people to take steps to make their lives better. A ship’s course is changed in small degrees over time.

When someone says “DevOps Doesn’t Work”, they’re absolutely correct. DevOps is a concept, a philosophy, a professional movement based in trust and collaboration among teams, to align them to business needs. A concept doesn’t do work, and a philosophy does not meet goals - people do. I encourage you to seek out ways of working better with your fellow people.

Gratitude

I’d like to thank my friends for listening to me rant, and my editor Shaun Mouton for their help bringing this article together. I’d also like to thank the SysAdvent team for putting in the effort that keeps this fun tradition going.

Contact Me

If you wish to discuss with me further, please feel free to reach out to me. I am gwaldo on Twitter and Gmail/Hangouts, and seldom refuse hugs (or offers of beverage and company) at conferences. Death Threats and unpleasantness beyond the realm of constructive Criticism may be sent to:

Waldo
c/o FBI Headquarters 
935 Pennsylvania Avenue, NW
Washington, D.C.
20535-0001

Day 2 - Examining Tools With a Devops Lens

Written by: Jennifer Davis (@sigje)
Edited by: Greg Poirier (@grepory)

Over the last few years I have had the opportunity to attend a variety of conferences, meeting people and hearing diverse stories about work environments and challenges. It is fascinating to hear the variety of explanations about devops and its ongoing impact on the industry and the workplace. As I worked to gain a better appreciation of others' experiences, I realized the challenge of misinformation about devops. How do we expect individuals to understand and choose devops when we have so many competing themes?

Two big themes I have heard:

• Devops is culture not tools.
• Devops is automation.

In this article, I will tackle one aspect of the challenge focusing on examining tools within the industry with a devops lens.

Devops is culture not tools.

What is culture? Culture is the totality of learned, socially transmitted customs, knowledge, objects, and behaviors. It includes the ideas, values, customs, and artifacts of a group of people. Culture encompasses everything about how we live and work, and how life and work evolve to meet the challenges we have in our environments. It gives meaning to social, political, economic, aesthetic, religious norms, and modes of organization that distinguishes us from others.

"We become what we behold, we shape our tools and then our tools shape us."

Father John Culkin

What are tools? Tools are a vital component of our identity, how we interact with the world around us, how we use and conserve energy, and how we buffer ourselves from harm. Tools shape our thoughts and behaviors including our transparency, level of exertion over our environment, and conflict response. Tools are a cultural artifact.

Consider the introduction of automobiles and the changes to city landscapes that came about because of them; new cities have wider streets to accommodate vehicles with less care taken to provide pedestrians with safe walking. Automobiles impacted how we work and where we work. As commute times have increased in densely populated regions, the value of owning a car has decreased and the landscape will transform again.

Prior to the introduction of devops, an organization might incentivize and reward teams based on each team's particular priorities; developers strove to build features quickly, operations stabilized systems, security minimized risk and controlled access, and so on across the organization. When the bonus for the VP of Operations is driven by uptime, system administrators will be incentivized to ensure stability and availability. When the bonus for the VP of Development is driven by meeting deadlines, developers will be incentivized to deliver features. This leads to a fundamental conflict in deciding highest priority between individuals trying to work together across organizational boundaries.

Organizations have created islands of culture based on their existing individual and collective values. For example, the roles and responsibilities of an operations engineer at one organization may significantly differ at another. Consequentially, operations engineers at different companies often have substantially disparate skill sets. Over time, the roles and concerns within an organization change leading to fractured capabilities even within a single organization.

As organizations grow larger, each team's individual focus narrows further. Each team adopts a specific tool that meets their requirements, leading to multiple tools that overlap in purpose. This proliferation of tools both hinders transparency within the organization and stifles collaboration.

For example, bug tracking within development teams led to the development of tools like JIRA and bugzilla. Ticket tracking within operations teams led to development of tools like Request Tracker(RT). Bugs and tickets are both issues. While a single issues tracking tool can be selected at the organizational level, the tool can impact the cognitive load of teams unequally as a team tries to adopt a tool that isn't specific to their domain.

The problem of tool proliferation continues today with the belief that a specific tool might solve all the problems an organization may have. When people say Devops is culture not tools, it is in part to focus on solving the organizational issues that can not be solved with a change in technology alone.

What is Devops?

Devops is an ideology that seeks to change how individuals think about work, value the diversity of work done, develop deliberate acceleration of business value, and measure the effect of social and technical change.

In Effective Devops, Ryn Daniels and I introduced the 5 pillars of devops: Collaboration, Hiring, Affinity, Tools, and Scale.

These 5 pillars form the foundation for effective devops and the devops compact, the continuous building of a shared mutual understanding within a team, organization, and across the industry.

With the devops compact, we commit to working together, communicate our intentions and any issues, and adjust our work in order to work towards shared organizational goals. We answer questions like: what is the value of what we are we trying to do, how are we going to do it, and what exactly are we doing? We don't just decide to do devops, and then we’re done. The nature of doing devops, means that we are commiting to adjusting to change.

Devops is culture which includes tools and their use. Tools are not devops. How tools are used, and the ease with which they can be used impacts the acceptance and proliferation of specific aspects of culture. When we talk about devops tools, we mean the tools and the manner of their use--not fundamental characteristics of the tools themselves.

Devops tools stress "We" over "Me"; they allow teams and organizations to build mutual understanding to get work done. Your choice of tools is a choice in a common language. Is this language one that benefits your organization as a whole or merely a subset of specific teams? At times, due to the lack of availability of an equally balanced tool, a choice must be made that has a higher cognitive cost to one team over another. Be aware of the cost and empathetic to the teams impacted.

The devops culture that we embrace is one of collaboration across teams, across organizations, and across industries. When developing solutions, I think about my team instead of thinking about the easiest thing for me to do now. This sometimes means adjusting my expectations for the good of the organization and choosing something a little more difficult than my current working structure.

Tool Examination

Let's examine a few tools with the devops lens. First we'll look at version control systems and then infrastructure automation. Please note that this examination of tooling is not meant as a judgement or advocacy of a specific tool in your environment. Without understanding your organization's culture, noone can tell you the right tool to choose.

Version control systems record changes to a set of files over time. CollabNet founded the Subversion project in 2000 as an open source software versioning and revision control system that was architected to be compatible with the widely used Concurrent Versions System (CVS). Subversion 1.0 (svn) was released in February of 2004. Technology and habits at the time dictated svn's use and features. Core to svn's architecture is the concept of a centralized repository. This central repository allowed users to control who was and was not allowed to commit changes.

A year later in 2005, Git was released. It’s also an open source software version control system with a focus on decentralized revision control, speed, data integrity, and support for distributed nonlinear workflows. This gives every developer full local control. While you can adopt a centralized workflow and establish a "central" repository, the processes can be flexible allowing you to use technology as you wish rather than having it defined for you. While the ramp up time may be a little longer, the functionality allows for quicker organizational changes.

Andy Peatling shared the WordPress technical refresh in migrating from svn to git in 2015, and the desirable changed behaviors that included:

• improved developer communication through code reviews and hangouts
• improved development practices of reviewing code prior to committing to the repository because of the availability of the pull request with git
• increased collaboration between designers and developers on a daily basis
• greater feedback on individual work,
• customized workflows

The technical merits between git and svn aside, significant value is derived from increased cross-team collaboration.

In many organizations, system configuration is a manual process. Individuals document the process and upgrade with a checklist. A missed step can lead to systems in an unknown state requiring considerable effort to recover.

Chef is an infrastructure automation system. Infrastructure automation is creating systems that reduce the burden on people to manage services and increase the quality, accuracy, and precision of a service to the consumers of a service.

When Adam Jacob was developing Chef software, he was trying to create a solution that could work across different organizations. Chef was built to provide abstractions for configuration and management, creating a language allowing individuals to define its infrastructure and policies with code.

Trying to create a language that allows for the nuanced views of developers, system administrators, security operations, and quality assurance engineers is difficult. With Chef, rather than reusing terminology that shows preference to one role over another, we have new terminology including resources and recipes.

Resources make up the basic building blocks of our infrastructure. We can use resources as provided by Chef core libraries, use resources provided by the larger Chef community, or we can build and customize our own.

Recipes describe a specific piece of an application that we want to have running on a system. It's the ordered set of resources and potentially additional procedural code. Just as with a recipe for baking chocolate chip or oatmeal cookies, the recipe will be specific to what we want to create.

In this simple webserver.rb recipe from the Learn Chef website, we use 3 resources: package, service, and file. These resources describe everything needed to stand up an apache web server, have it run at boot time, and have it serve a simple "hello world" page.

More complex tasks like opening specific ports in a firewall or deploying Hadoop can be abstracted into a common language that allows for comprehension across teams. Because this code is stored in your standard version control, you can take advantage of the same ease of use and transparency that the rest of your source code is afforded.

Devops is not limited to infrastructure. A lot of focus is on tools that improve collaboration between development and operations teams as those teams are where the devops movement originated. As other teams begin adopting devops methodology, we'll see tools that support the devops compact between new teams. For example, Chef has released Chef Delivery this year. Chef Delivery simplifies organizational complexity by creating a common language to describe the process of promoting products between teams. This common language allows us to codify "delivery as code".

Devops is Automation

When people say Devops is Automation, it is due in part because many tools in devops, while codifying understanding to bridge the chasm between teams and increase velocity, have resulted in automation for repetitive tasks. Automation is a result of improved technology focused on repeatable tasks.

While reviewing the aviation industry in 1977, the House Committee on Science and Technology identified cockpit automation as a leading safety concern. Studies of the aviation field discovered that while pilots could still fly planes, the automation in use was causing an atrophy to critical thinking skills. Pilots were losing the ability to track position without the use of a map display, decide what should be done next, or recognize instrument system failures. This is a warning for us as we implement automation within our environments. Tools change our behavior and the way that we think.

In July 2013, Asiana Airlines flight 214 struck a seawall at San Francisco International Airport, resulting in three fatalities. During the investigation, the National Transportation Safety Board (NTSB) identified a number of issues, among them that there was insufficient monitoring of airspeed due to an overreliance on automated systems that they didn't fully understand.

“In their efforts to compensate for the unreliability of human performance, the designers of automated control systems have unwittingly created opportunities for new error types that can be even more serious than those they were seeking to avoid.” James Reason, Managing the Risks of Organizational Accidents

Automation is critical to us as systems become more complex and organizations become interdependent due to shared services. Without shared mutual context or concern for human needs, automation creates unknown additional risk.

Devops enabled automation may make work faster, but more importantly it has increased transparency, collaboration, and understanding.

In summary, devops is culture which includes tools and their use. Tools help us progress from manual to continuous processes. Effective devops tools enable automation without isolating humans from the automation process so that the mutual understanding occurs across time, ensuring that the "we" of tomorrow understands the "we" of today.

What Next?

Find a local DevOpsDays, CoffeeOps, or other cross-functional meetups that facilitates people coming together to build strong foundations of common understanding.

The devops community calendar also lists upcoming conferences and events.

Study

• Github git tutorial
• Atlassian Git tutorial
• History of Chef
• Learn Chef

Watch

• DevOpsDays Silicon Valley 2015 Videos
• Tools for Effective Change Webinar

Listen

• Arrested DevOps
• Foodfight Show
• DevOps Cafe

Read

• devops Weekly - weekly newsletter of curated devops news from Gareth Rushgrove
• Comparison of Issue Tracking Systems
• Comparison of Version Control Software
• NTSB report for Asiana Airlines flight 214
• Effective Devops

Share

I'd love to hear your feedback! What tools are you using in your environment and how do they impact organizational culture?

Tweet at @sigje with hashtag #sysadvent, email me at sparklydevops@chef.io, or add a comment to the article.

Thank yous

Many thanks to Greg Poirier for being my Sysadvent Editor. Additional thanks to Nathen Harvey, Peter Nealon, Amye Scavarda, and H. Waldo Grunenwald for peer review and aditional edits.

• Image Credits
  • Young Girl Tilting Camera Up. Licensed from Kenneth Wiggins on https://blackstock.co/collection/young-girl-tilting-camera-up/
  • Light Sky Silo Windows Sunlight Rays from Public Domain Archive

Day 24 - 12 days of SecDevOps

Written by: Jen Andre (@fun_cuddles)
Edited by: Ben Cotton (@funnelfiasco)

Ah, the holidays. The time of year when we want to be throwing back the eggnogs, chilling in front our fake fireplaces, maybe catching a funny Christmas day movie… but oh no we can’t, because guess what, a certain entertainment company was held hostage by a security breach the likes of which corporate America has never seen before… and no more movie for you.

It’s an interesting time to be a security defender. The recent Sony breach has just put a period on the worst-of-the-worst scenarios that us tinfoil-hat, paranoid security people have been ranting about all along: one bad breach could be business shattering.

But let’s step back, and look at the theme of this blog: the 12 days of SecDevOps. Besides being a ridiculous title that I’m 90% sure my ops director chose specifically as a troll for me (thanks, Pete), it underlines an important concept. Whether `security` is in your job title or not, operations is increasingly becoming the front-line for implementing security defenses.

Given that reality, and the fact that security breaches are NOT going away, and that most of us don’t have yacht-sized security budgets, I thought it would be interesting to come up with 12 practical, high-impact things that small organizations could be doing to shore up their security posture.

Day 1: Fear and Loathing and Risk Assessment and Hipsters

Risk assessment. It’s not just some big words auditors love to use. It’s simply weighing the probability of bad things happening against the cost to mitigate the risk of that bad thing happening. And using that to make good security decisions as you make day-to-day architecture and ops choices:

risk = (threat) x (probability) x (business impact)*

*whoever told you there would be no math lied to you

You may not be aware of it, but as an ops person you are likely doing risk assessment already, except more likely around things like uptime and reliability. Consider this scenario:

• John, the web guy, proposes replacing PostgreSQL with SomeNewHipsterDB.
• You ask yourself, ‘huh, what’s the chances that I’m going to get paged at 3am because writes stop happening and my web site starts screaming in pain?’ You are probably not having warm-fuzzy feelings about this plan.
• Your development and ops team evaluates the benefits to the engineering team and business for switching to SomeNewHipsterDB and weighs it against the probability that you are going to get woken up all of the time, and the impact it will have on your sunny disposition and decide that yeah… maybe not gonna do it.
• Or, you do, except you mitigate this risk by saying ‘John, you will be forever paged for all SomeNewHipsterDB issues. Done.’

Cool. Now do this for security. Every time you are making architecture choices, or changing configuration of your infrastructure, or considering some new third-party service SaaS you’ll be sending data to, you should be asking yourself: what’s the impact if that service or system gets hacked? How will you mitigate the risks?

This doesn’t have to be a formal or fancy report. It can be a running text file or spreadsheet with all of the possible points of failure. Get everyone involved with thinking of ways pieces of the infrastructure or organization can be hacked, and ways you are protected against those worst-case scenarios. It can be like ‘ANYONE WHO OWNS OUR CHEF SERVER COULD DESTROY EVERYTHING [but we have uber-monitoring and Jane over there reviews audit logs daily]’. Start with the scenario: what if… ? and have conversations with engineers and business owners defend why what we’re doing is good enough. Make security a fundamentally collaborative process.

Day 2: Shared Secrets: Figure it Out Now

There’s 3 things in life that are inevitable: death, taxes… and the fact that a sales guy left to his own devices will always put all of his passwords in a plain text file (or if fancy, an Excel spreadsheet).

The lesson is this: password management isn’t something that just the technical team decides and manages for itself. We should be advocating organization-wide education on managing credentials, because guess what? Access to Salesforce, Gmail, and all of these SaaS services with sensitive business data are being used by people who are not engineers.

Solution? As part of every employee’s onboarding process, install password management on an employee’s workstation, and show them how to use it (e.g. 1Password or LastPass, or whatever your tool of choice is). Start doing this from the outset, as it’s best to figure this out on Day 1 rather than 200 employees in.

Day 3: Shared Secrets for Infrastructure, Too

When it comes to infrastructure secrets, there are extra concerns because in most cases, systems needs to be able to access these secrets in a non-interactive, automated way (e.g. I need to be able to spin up an app server that knows how to authenticate to my database).

If all of your infra passwords start unencrypted somewhere in a git repo, You Are Going To Have A Bad Time. Noah has a good article on various options for managing shared secrets in your infrastructure.

Day 4: Config Management On All Of The Things (So You Aren’t Sweating from Shell Shocks)

This should be obvious to everyone who drinks from the DevOps Koolaid, but CM has done beautiful things for patch management. It may be tempting to deploy a one-off box used for dev manually without config management installed, but guess what? In the case of Browser Stack, that turned out to be a massive achilles heel.

Making the process easy for devs to get access to the infrastructure they need (while giving you the ability to manage systems) is key. Do this right away.

Day 5: Secure your Development Environments (Because No One Else Will)

If left to their own devices, development environments tend to veer to chaotic. This isn’t just because developers are lazy (and as a developer, I mean this in the nicest possible way) but because of the nature of the prototyping and testing process.

From a security perspective, this all means bad juju (see Browser Stack example above). I can assure you that if you start building your prototype or dev infrastructure exposed to the public internet, deploying it without even the basic config management, it will stay that way forever.

So: if you are using AWS, start with an Amazon VPC with strict perimeter security, and require VPN access for any development infrastructure. Get some config management on everything, even if it’s just for system patches.

Put some bounds around the chaos early on, and this will make it easy to mature the security controls as the product and organization mature.

Day 6: 2-Factor all of the things (well, the important things)

Require 2-factor wherever you can. Google Apps has made enforcing this super easy, and technologies like DuoSecurity and YubiKey make adding 2-factor to your critical infrastructure (e.g., your VPN accounts) far, far less annoying than it used to be.

Day 7: Encrypt your Emails (and other communications)

Encrypt your emails. It’s annoying to set up, but guess what? Hackers just love to post juicy stuff on pastebin. Again, from Day 1, help every single employee configure PGP or SMIME encryption as part of the onboarding process. Once installed, it’s relatively painless to use (as long as you don’t mind archaic mail clients from 1999).

This is especially important to drill into executives because they tend to have more sensitive emails (e.g. their private boardroom chatter), and are particularly susceptible to phishing style-attacks. With the recent Sony email leaks, you now have some leverage. You can throw the ‘Angelina Jolie’ emails in front of them and ask: how much do you think business and reputations would suffer were their entire email archives publically disclosed via a breach?

For many of us, chat is as crucial as email in terms of the type of reputation-critical information we put there. It may not be reasonable to switch to a self-hosted chat solution, but in that case, ensure you are picking a service that helps YOU mitigate your risk. E.g., do you need all of the history? Do you need private history for user chats?

Day 8: Security Monitoring: Start Small, Plan Big

Put the infrastructure in place to collect as much security data as possible, then start slowly making potential security issues visible by adding reports and alerts that deal with threat scenarios you are most worried about.

Start small. Remember that risk assessment list you made? Identify what you are most afraid of (um, that PHP CMS that has hundreds of vulnerabilities reported per year? Your VPN server?) and tackle monitoring for those items first.

Instrumenting your infrastructure from day 1 for security monitoring (even if it’s just collecting all of the system and application logs) puts you in a good position later on to start sophisticated reporting and intrusion detection on that data.

Day 9: Code/Design Reviews

Although there have been a lot of advancement in static and dynamic source code analysis tools (which you can integrate right into your CI process), a good-old fashion code review by a human being goes a long way. If you’re using GitHub, just make it part of the development workflow and testing pipeline. Whenever changes are made to authentication or authorization, have someone look for automated tests that deal with those cases.

Day 10: Test Your Users

Phish yourself regularly. It’s really easy to do, and can be illuminating to the rest of the business which may not be as technical as the operations/engineering side, and not really understand really the impact of opening an attachment in an email or not checking URLs where they are logging into a website. You can use some open source tools, but are also many services now that you can pay to do this for you.

Day 11: Make an Incident Response Plan Now

So, you see something odd in your logs. Like, Bob your DBA ran a Postgres backup on production DB, tar’d it up, and sent it to an FTP server in Singapore. Bob lives in Reston VA, and this is definitely not normal. You start seeing evidence of other weird stuff ‘bob’ is doing that he shouldn’t be.

What now? Do you email Bob and say ‘something weird is happening?’ Do you call the Director of Ops? Do you put a message in a lonely chat room?

Figure out a plan for escalating possible critical security issues. Doesn’t have to be fancy or use specialize ITIL incident response workflow tools. Make a group in PagerDuty. Have an out-of-band channel for communicating details, in case your normal network goes the way of Sony and is totally compromised or just plain is not working. Maybe it’s as simple as an email list that doesn’t use the corporate email accounts, or a conference bridge everyone can hop on.

Day 12: Don’t be the Security ‘A**hole’

You. Yes, you. Don’t be the security a**hole that gets in everyone’s way and loses sight of the real reason for everyone’s existence: to run a business. You can be the security champion without being the blocker. In fact, that’s the only way to be effective. If a user is coming to you and saying ‘this is really really annoying, I don’t want to do it’ - listen to them. Too many security personnel disregard the usability issue of security controls for the sake of security theater, which leads to (unsurprisingly) abandonment, cynicism, and apathy when it comes to real security concerns.

DevOps is really a philosophy: it’s not a job title, or a set of tools, it’s the concept of using modern tools and processes to facilitate collaboration with the engineers who deliver the code and those who must maintain it. Um, that was a lot of words, but the key word is collaboration. It’s no longer acceptable to throw ‘security over the wall’ and expect your users and ops people to just do what you say.

The best security cultures are not prescriptive, they are collaborative. They understand that business needs to get done. They are intellectually honest and admit ‘yeah, we could get hacked’ - but what can we do about this in a way that doesn’t bring everything to a halt? Zane Lackey has a great talk on building a modern security engineering organization that expounds many of these ideas, and more.