Day 6 - sysadmins - the evolution of a role amidst revolutionary hype.

By: Robert Treat (@robtreat2)

Edited By: Daniel “phrawzty” Maher (@phrawzty)

Like so many things in our industry, our job titles have become victims of the never-ending hype cycle. While the ideas behind terms like “DevOps” or “Site Reliability Engineering” are certainly valid, over time the ideas get lost and the terms become little more than buzzwords, amplified by a recruiting industry more concerned about their own short-term paychecks than our long-term career journeys. As jobs and workers are rebranded as DevOps and SREs, many Ssysadmins are left wondering if they are being left behind. Add in a heavy dose of the cloud, and a sysadmin has to wonder whether they will have a job in a few years. The good news is that despite all the noise, the need for sysadmins has never been stronger, you just need to see the connections between the technology you grew up on, and the technology that is going to move us forward in the next several years.

It used to be that when you started a new project you first had to determine what hardware to use, both from a physical standpoint but also from a pricing standpoint. In the cloud, these concerns are still there but have shifted. While most people in the cloud no longer worry about sizing infrastructure correctly at the start of a project, the tradeoff of being able to re-size VMs with relative ease is that it is also easy to oversize your instances, and in the cloud oversize means overspend, all day every day; at some point you need to work out the math to determine what your growth curve looks like in terms of resource needs vs dollars spent. Ever made that joke about having to bust out the slide rule to run cost comparisons between Riverbed, NetApp, and LSI? As much as they try, the cloud hasn’t made IOPS go away. Helping set estimates on how many IOPS an application will consume still requires a bit of maths, only now you also need to know your way around EBS and SSDs vs Provisioned IOPS in order to determine a reasonable IOPS per dollar ratio. But hey, you’ve done that before.

And that’s the thing; there are many skills which transfer like that. Scared about the new world of microservices? Don’t be - you were dealing with microservice like architectures long before the rest of us had even heard of the term. At it’s core, microservices are just a collection of loosely coupled services designed to meet some set of business goals. While we have not traditionally built and run applications that way, the mental leap for a sysadmin familiar with managing machines running Apache, Sendmail, OpenLDAP, and Squid is much less than for a developer who has only ever dealt with building complex monolithic applications. As sysadmins, we don’t think twice about individual services running on different ports, speaking different protocols, and providing completely different methods for observing their behavior; that’s just the way it is. Compare that to a development community that has wasted a generation trying to build ORMs to abstract away the concept of data storage rather than just learning to talk to another service in its own language.

This isn’t to say you can rest on your laurels. The field of Web Operations and the software that powers it is constantly changing, so you need to develop the ability to take what you already know and apply it to new software and systems. It is worth pointing out that this won’t always be easy or clean; new technology often misrepresents itself. For example, the rise of tools like Chef and Docker left many sysadmins wondering which direction to turn, but if you study these tools for a bit, you see that they draw similar patterns to old techniques. It can certainly be difficult for folks who have spent years coding on the command line to grok the syntax of a configuration management tools DSL, but you can certainly understand why companies want to automate systems; the idea of replacing manual labor with something automated is something we print on t-shirts just for fun. And sure, I understand how the yarn ball of recipes, resources, and roles might look like overkill to a lot of people, but I’ve also seen some crazy complex mixes of bash and Perl used as startup scripts during a PXE boot, so it’s all relative.

When Docker first came on the scene, it also promised to revolutionize everything we know about managing systems. All the hype around containers seemed to focus on resource management and security, but the reality was mostly just a new way to package and distribute software, whereby new I mostly just mean different. I’ve often lamented that something is lost when an operator never learns how to compile code or bypasses the experience of packaging other people’s software. The promise of Docker is that you can put together systems like using a set of legos using pre-existing pieces, but stray from the well trodden path even a little and you’ll find all of those magic compile time errors and strange library dependencies that you are familiar with from systems in the past. Like any system built on abstractions, there are assumptions (and therefore dependencies) baked in three levels deep. If you ever debated whether to rewrite an rpm spec file from scratch after a half day hacking on the distro’s spec file trying to add in the one module you need the maintainers didn't… replace rpm spec file with dockerfile and you have someone to share root beers with. Sure the container magic is magic when it works, but the devil is in the dependencies.

Of course no conversation about the role of the sysadmin would be complete without touching on the topics of networks and security. While sometimes made the purview of dedicated personnel, at some level these two areas always seem to fall back to the operations team. Understanding the different networks within your organization, the boundaries between those networks, and the who or how to traverse them has always been a part of life as a sysadmin. Unfortunately in what should be one of the most directly applicable skillsets (networks are still networks), the current situation in cloud land has actually gotten worse; the stakes are fundamentally higher in a world where the public internet is always just a mis-configuration away. Incorrect permissions on a network file share might expose sensitive material to everyone in the company, but incorrect permissions in S3 expose those files to the world. Networking is also more complicated in the cloud world. I’ve always been pretty impressed by how far one could get with VPNs and SSH when building your own, but with cloud providers focused on attracting enterprise clients in regulated industry, you’ll have to learn new tooling built to meet those needs, for better or worse. It can still be done, just be aware it is going to work a little differently.

So the good news is that the role of the sysadmin isn’t going away. While the specifics may have changed, resource management, automation, packaging, network management, security, rapid response, and all the other parts of the sysadmin ethos remain critical skills that companies will need going forward. Unfortunately that doesn’t solve the problem of companies thinking they need to hire more “DevOps Developers” (though I never see jobs for “DevOps Operators” - go figure!) and other such crazy things. As I see it, it is easy to make fun of companies who want to hire DevOps engineers because they don’t understand DevOps, but you can also look at it like hiring a network engineer or security engineer - believing you need someone who specializes in automation (and likely CM or CI/CD specifically) is not necessarily a bad thing. So the next time you’re feeling lost or wondering where your next journey may take you, remember that if you focus on the principles of running systems reliably and keep your learning focused on fundamental computing skills, even though the tools may change the problems are fundamentally the same.

Day 15 - Fear and Loathing in Systems Administration

Written by H. “Waldo” Grunenwald (@gwaldo)
Edited by Shaun Mouton (@sdmouton)

“DevOps Doesn’t Work”

The number of times that I’ve heard this is amazing. The best thing about this phrase is that the people who say it are often completely right, even if for very wrong reasons.

Who Says This?

Well, let’s talk about the people who most commonly have this reaction: SysAdmins. I’m going to use the term “SysAdmins” as a shorthand for a broad group. The people in this group have widely varying titles, but it is most commonly “Systems”, “Network”, or “Operations” follwed by “Administrator”, “Engineer”, “Technician”, or “Analyst”.

In some companies, these folks have the best computers in the place. In others, they have to live with the worst. Their workspace probably isn’t very nice, and almost certainly has no natural light. If there is a pager rotation, they are almost certainly on it. If there isn’t a rotation, they’re basically on-call all of the time.

During the course of a normal day they might have to switch contexts between disaster planning, calculating power and HVAC needs for a new datacenter, scrambling to complete an outage-driven SAN migration, rushing to address urgent requests to help people with their email, to troubleshoot a printing problem, or suss out why someone can’t get to their electric company’s bill pay website. They may be the sole person with database expertise in the company, or they may work on a team of dozens.

The work is largely invisible except when something fails, in which case it’s highly visible and widely impacting.

These are typically cynical people, because there are only so many times that you can’t make the team/department/company party for ostensibly “celebrating our successes” because something’s broken, and you’re left to clean up after the “success”. There are only so many times that one sees a new project announced and begins to hire more people. When asked who’s going to support the new project, the response is a blank look and “you are”. The “…of course.” may not be vocalized, but it’s probably there. When asked how many people they get to hire to help with the workload, the response is a combination of “sorry, but there wasn’t anything left in the budget”, “it won’t be that much more work”, or a variation of the “team player / good soldier” speech. There are only so many times one can take getting your requests for training or conference budget rejected out of hand, and have your requests for training or conference budget laughed out of the room.

They probably have basic working knowledge of a half-dozen programming languages, but most likely they often think in Shell. They probably know at least three ways of testing if a port is open, and probably have a soft spot in their heart for a couple of shell commands.

They may have seen or participated in a DevOps initiative that consisted of a team or position rename, or helpfully suggested that they install some Config Management and Monitoring software so that “we can DevOps now…” or “so we can do Agile”. When they hear “DevOps” or “Agile”, what they are hearing is is Let’s take the same people who can’t handle a planned release schedule or make whatever effort that they need to squeak by the Change Board and Release Management requirements, and give them unfettered access to Production. Clearly, I’m not paged often enough.

So what is one to do? How is one to maintain their sanity in the face of increasing job scope, increasing demand for access and velocity, and little hope for an effective new-hire count? Not to mention continuing to juggle the existing volume of requests, and continuing to grease the existing gears to keep the machine running.

Get Help

Please note that I’m not saying “just”. There’s nothing just about this situation; there is nothing simple about any of this, and Justice hasn’t been seen in a long time in an environment where this is the norm. Most of these changes are difficult. They will take work, and will require convincing other teams to join in your cause.

Admitting you have a Problem

The problem (probably) isn’t technical. It’s almost entirely social.

Because SysAdmins are typically responsible for the environment, the easiest way to assure that the state is stable is to lock everyone else from it. While this helped with the goal of “keeping out unexpected changes”, it had a number of side effects.

First, a kind of learned helplessness has set in. Your customers and teammates became so used to being “hands-off”, that they don’t have the wherewithal to meet reasonable expectations. Since they’re uncomfortable making any changes, all changes must be made by the SysAdmins. This leads to your time being taken by having to perform lots of low-value tasks.

Some teams settle on the pattern of “hands off Production, but you have access to Staging”, but this is fraught with peril. The most common problem that stems from this is “Configuration Drift.” Config Drift is when you have different settings in one environment (or server) than the others. When the cost to discover what Production looks like is high, it’s more likely that people will either use defaults, make assumptions, or use the same configs that they use in their IDEs. “Works on my machine”, indeed.

This is a problem well-solved by Configuration Management tools, but you still need to be willing to trust your peers and give them access. If you want to be part of the process of validating changes, you could put in place the structures that allow a pull-request and code-review workflow, something that your Software Engineering peers should be very accustomed to! Granting access to see the existing configs and the ability to propose changes also shares responsibility for your team’s environments and contributes to feelings of ownership. Denying colleagues the ability to effect necessary configuration changes contributes to the root problems of configuration drift and learned helplessness.

Stop Feeding the Machine

Your value is not in doing the work, but rather being able to make the decision to do the work.
I’ll be the first to say that “Automating ALL THE THINGS” is a flawed goal. At work, it’s usually said in the context of a Project, rather than part of a philosophy of Continual Improvement (Think Toyota Kata). You shouldn’t have to engage in an “Automation Project” to improve your environment. Build into your schedule time to solve one problem. Pick something that is rough, manual, and repeatable. Remove a small piece of friction. Move on to the next one. Hint: Logging into a server to make a configuration change should be a cue to implement configuration management!

While I agree that everything being automated, not everything should be automatic. Decision-making is complex, and attempting to codify all of the possible decision-making points is a fantastic way to make yourself insane. Not to mention that documenting your decision-making processes may be an unwanted look inside your brain. Caveat Implementor. (Or perhaps that’s just me…)

All of the units of work should be automated. But the decision to run the now-automated tasks can be left to a human. When you find that there is a correlation between steps, those pieces should be wrapped together. Automation isn’t a project into itself. It should be iterative. Pick something that’s painful. Make it a little smoother. Repeat. Ideally, you have time blocked out for Continuous Improvement. If not, create a meeting, or create a weekly project to do so. Review the issues that you’ve experienced lately, and pick something to make better. It might be worth making into a project, but it won’t be an ALL-THE-THINGS project. Create a scope of effort. Take the time to plan goals and requirements.

Whatever you don’t automate must be documented. Beyond the typical benefits of documentation, it also serves as “Functional Requirements” for someone else to pick up when they can help you with providing a solution. Try to recognize whether documenting or automating takes longer. Perhaps this piece of documentation will bet better served by “Executable Documentation” (i.e. code).

Clarify Your Role

You should attempt to pick apart the parts of your work, and attempt to describe them. One way to make this a fun exercise is to use other job titles to describe the work.

Are you an “Internet Plumber”? How much of your job could be described as “Spelunking” into the deep dark caverns of Legacy systems?

If you want, you could ascribe Superhero names to these parts of your work. The added bonus is that it not only describes a role, but also a demeanor associated with them. When ‘bad code’ makes it to Production, do you go “Wolverine” on that dev team?

Could you describe part of your role as “Production Customs Official”? Are you the gateway to Production? If so, are you actually equipped to do that? Here’s a quick test: When you say “no, that can’t go”, do you get overridden?

More importantly, is this what you want to do?

Prepwork

You will need to prepare for this. Most SysAdmin teams do not have a healthy relationship with the rest of the business. You will need to initiate the healing.

Take someone to lunch. Preferably someone who you don’t know well. Ask questions, and listen to the answers. It is not time to defend yourself or your team. It’s time to find out what the business needs from someone else’s perspective. Ask what they think that your team’s role is in toward achieving that success. Ask what they think your team does well, and where there are gaps between what you have now and excellence.

Speak their language

You probably recognize their words, but you need to go out of your way to speak them. To communicate your message, you will need meet them on their turf. This may seem terribly unfair - “Why can’t they meet me on my terms?!” - but I’m guessing that has not been working out well for you so far.

Not only do you need to use their language, but you need to communicate over their medium. And identifying who they are is step one in learning to speak it. It’s probably not IRC, and only writing it in email is a good way for it to be ignored.

If you’re speaking to management, be prepared to write a presentation. Executives especially like to see a slide-deck. It doesn’t have to be slick. It probably shouldn’t have sounds or much in the way of transitions, but a presentation can help to lay the groundwork for a conversation.

Discuss Scope, Staffing, and Priorities

Now that you have described your role, we also need to describe everything that you support.
What Products do you support? It’s entirely possible (likely, even) that the people and teams that you support don’t actually know what you’re responsible for. It could be argued that most of them shouldn’t need to know. But if you have been saying “no” to protect yourself, it’s a sign that you are significantly overextended. You need to have a real discussion with your leadership about your role, scope, and staffing.

In order to have this discussion, you need to prepare. You need to come up with a fairly comprehensive list of the products and teams that you support. This is a list of every team, and their products, the components and tasks that belong to you for each. Don’t forget all of the components that “nobody owns” but somehow people come to you to fix or implement (CI, SCM, Ticketing, Project, and Wiki tools seem to be common examples). Are you also responsible for Directory Services? Virtualization platform? Mail/Chat/Phones? Workstation Purchasing and provisioning? Printers? Do you manage the Storage, Networking, etc? Don’t be afraid of getting into details. It can help to provide clearly written potential impacts the company if some of these “hidden” services stop working? Your leadership might not know what LDAP or Directory Services are, but they’ll understand if nobody can log into their machines, they can’t pull information to build reports, and by-the-way nobody can deploy code because it relies on validating credentials…

What is most important to the company? What do you need to succeed? How much more staff do you need? What tooling or equipment would help you work more efficiently? Does code deploy even when it fails testing? How many outages have arisen due to this happening?

Demonstrate Cost and Value and Revisit Priorities

In order to have meaningful discussions with people in your company who aren’t necessarily technical, you need to be able to relate to a language that they speak. Regardless of team duties, the lingua franca of most teams is money. As Engineers, most of us prefer to think in terms of the tech itself, but in order to describe an impact, a unit of monetary value is a proxy for impact that most non-technical people can understand, even if they don’t grasp the details.

It is a helpful (if difficult and uncomfortable) habit to get into, but I encourage you to consider the components of cost that goes into every incident or task.

What is the cost of a main-site outage? How much revenue does this feature bring in? Why are you spending so much on infrastructure and effort to make that component Highly-Available? Why does it matter that you do that piece of maintenance? Show the negative value of doing things they way they are (Opportunity Cost), versus investing time to improve the automation around it. Describe how doing this maintenance work reduces your context switching, unplanned outages, and lost reputation of your company. Describe the benefit in increased visibility to the business, and Agency to be gained by your peers on other teams.

Why put in place these tools to let product teams self-serve? Describe that the features that the company’s teams spend so much time and effort (read: “money”) creating means nothing if those features aren’t available for customers to use. That having those features not available costs money in terms of feature billing, and reputation cost. If they claim that they’re doing Agile, but can’t do Continuous Delivery, they’re not really Agile, and the whole point of that framework is to improve delivery of value to the customer and the business!

Further, show how systems relate. It doesn’t have to be terribly detailed. Describe that the features that the customers use are reliant on x, y, and z components of infrastructure. Draw the lines from LDAP to storage to your CI tool to testing code to artifacts delivered to Production. Then show some of the other systems that have similar dependencies.

Once the picture emerges showing how everything is reliant on unexciting things like LDAP, your Storage cluster, and that janky collection of angry shell and perl scripts that keep everything working, realization will begin to dawn.

Congratulations, you’ve just effectively communicated Value.

Align Responsibility with Authority

Are you held responsible for apps written by other people? Who gets paged when “the app” goes down? How does that make sense?

Get Devs on-call for their apps. SysAdmins should be escalated to. Devs can triage and troubleshoot their own apps more readily than you can. They get to call in the cavalry when they get stuck. They don’t need to know everything about the systems, and they don’t need to resolve everything. When a fault occurs and they need help, they stay on the call, pairing with you as you diagnose, troubleshoot, and resolve. That way, they don’t need to escalate to you for that thing the next time it occurs, and can collaborate on automating a permanent fix.

When teams aren’t responsible for their products - When they aren’t paged when it fails - they are numb to the pain that they inflict. They’re not trying to cause pain; they just don’t feel it. It’s especially easy to argue this for teams that proclaim that they use Agile development methods: If they claim to want “continuous feedback”, there is nothing more visceral for providing feedback than the feeling of being awoken by a pager in the middle of the night. When the inevitable exclaimation comes that “we can’t interrupt our developers”, ask if it makes sense to interrupt someone else.
Even being aware of the pain (say, hearing how many times you were paged last night) can elicit sympathy, but that’s a far cry from the experience of being paged yourself.

Further, this is what that list of responsibilities is for. Asking each team to take responsilbity for their own products, you will still likely have a hefty list of services that you provide that you are on-call for. As these set in, point out the staffing numbers. This may be a matter of the places that I have worked, but I have never seen a Developer-to-SysAdmin ratio of less than 5-1. In most places it is much higher. By adding these teams to pager rotations, they drastically reduce the load on you. By not adding them to pager rotations, they are complicit in your burnout.

Stop saying “No”

SysAdmins have a reputation for saying “No”. The people who are asking are probably not trying to make your life worse; They’re probably just trying to get their work done. They might not know what their “simple request” involves, and that it probably isn’t necessary.

But by not having Responsibility aligned with Authority, you may have been stuck with the pain of other people’s wishes. You know that fulfilling their request will cause you pain, so understandably, you say “no”. What often happens next is that they escalate until they hit someone sufficiently important enough to override you.

This is the basis for why SysAdmins feel steamrolled by everyone else, and everyone else feels held hostage by SysAdmins.

But all hope is not lost.

Stop saying “No”.

“Yes, but …” is a very powerful thing.

“Yes, but …” can be used to get you help.

“Yes, I can set that up for you, but we don’t have capacity to run it for you.” What happened there? You agreed that the request is reasonable. You set expectations of the level of support that you can give. You left the requestor with several options to continue the conversation.

• They might have hiring reqs that they can’t fill. You can negotiate for some of them to go to your team, as you’re clearly understaffed.
• Some of their engineers may join your team as a lateral move. They’ll need mentorship and training, but this kind of cross-training is invaluable. It’s a force multiplier. It also sets precedent.
• They might take the responsibility for the Thing. They run it. They get paged for it. Of course you will probably have to be an escalation point to assist when it fails, but it’s their product. This again sets precedent.
  

Delegate

Most SysAdmins are stuck doing tasks that provide very little value because they restrict access to their peers. To my mind, there is one perfect example: “Playing Telephone”.

When I say “Playing Telephone”, I’m talking about the situation where someone (let’s say a Developer) wants logs from the application, but they don’t have access to get them. They request the logs from you. You fetch the log requested and provide it to them. “No, not that log, this log…” You fetch. “Hmm, I’m not seeing what I’m looking for, could you check in here for something that says something like this …?” And so on, and so on…

I don’t know what you’re hoping to prevent by restricting access, but if this scenario ever happens, you should know that you’re providing Negative Value. Again, let’s try to remember that your peers are not out to get you, and can probably be trusted to be reasonable humans if you meet them mid-way.

With that framework in mind, it’s time that you demonstrate some trust, and Delegate to them. Give them access. Your value is not in the logon credentials that you have, otherwise you’re just a poorly-implemented “Terminal-as-a-Service”.

Even better than giving access, is giving Tooling. Logging into a server should be an antipattern for most work! You need some better tooling. So, with the example of logging, let’s talk tooling.

Logging

First, logging into boxes to get logs is just dumb. Sure, you could wrap a tail command in a Rundeck job, but let’s Centralize those logs while we’re at it.

SysLog is better than nothing, but not by much. Shipping logs is easy, but consuming them as something useful is not. Batteries not included.

If your company wants to spend the money on Splunk, then encourage that. Splunk is a fantastic suite of tools, but I might wave you away from it if you’re not going to use it for everything. It’s going to be expensive, and if you’re not going to spend enough to use it for everything, there will be confusion as to what’s in there, and what’s stored elsewhere.

ELK (ElasticSearch + Logstash + Kibana, sometimes mistakenly simplified to “Logstash”), or “Cloudy Elk” / “ELK-as-a-Service” is a good middle-ground. ELK is Free (as-in-beer), and very featureful.

Take your Centralized logging of choice, and provide your customers with the url to the web interface. Send them links to the “How to use” docs, and get out of their way!

Terminal-as-a-Service

If someone asks you to “run this command for me”, you need to put a button on it.

You don’t need to RUN-ALL-THE-THINGS!

Rundeck is a fantastic tool to “Put a button on it”. Other people use their CI tools (like Jenkins or Bamboo) for this. My friend Jeremy Price gave an Ignite Talk at DevOpsDays NYC 2015 that describes this.

Personally I like Rundeck, because it’s pretty easy to make HA, tie it into LDAP for credentials, manage permissions, and by shipping it’s logs (see what I did there?), you get Auditing of who ran what and when!

If you have some data that Must be restricted, try to isolate those cases from the rest of your environment. You shouldn’t have to restrict Everything just because Something does need isolation.

Deploying Code. Yes, to Production

Why would you want to have to deploy other people’s code?! Do you really provide any value in that activity? If the deployment doesn’t go well, you’re launching another game of “Telephone”.
What if you make it easy for them to do it? Empower them with trust and tooling, making it easy to do the right thing! Give them tooling to see that the deploy succeeded! Logs are a start, but Metrics Dashboards that show changes in performance conditions and error rates will make it plain to see if a deployment was successful!

This Freedom doesn’t come free. Providing tooling doesn’t absolve the development teams of the need to communicate; in fact, it’s likely that they’ll have to communicate more. They will need to be watching those dashboards and logs to see for themselves the success of every deploy. They will also be more readily on-hand to help triage the inevitable instances when it doesn’t go swimmingly.

Us

I say “They” in this article a lot. And that is because, by default, most organizations that I have been a part of or heard stories of have had a strong component of “Us-Versus-Them.” It’s only natural for there to be an “Us” and a “Them”, but thinking in those terms should be a very short-term use of the language. Strive for the goal of a “We” in your interactions at work, and reinforce that language wherever possible. While it may not be My job to do “foo”, it is Our job to ensure the team and company is successful.

While that may sounds like some happy-go-lucky, tree-hugging, pop-psychological nonsense (and it is…:), the goal here is to get you, the beleaguered SysAdmin the help that you need, in order to improve the capabilites of the business.

Coda

There is so much more to this topic, particularly the shift away from a Systems team supporting a bunch of Project teams to a series of largely self-sustaining Product teams, but that will have to wait for another day.

The psychological damage done to SysAdmins by their peers can make us bitter and cynical. I encourage my people to try to see that “They” aren’t trying to make life difficult for you, but it’s very likely that Authority and Responsibility are misaligned. I likewise encourage my people to take steps to make their lives better. A ship’s course is changed in small degrees over time.

When someone says “DevOps Doesn’t Work”, they’re absolutely correct. DevOps is a concept, a philosophy, a professional movement based in trust and collaboration among teams, to align them to business needs. A concept doesn’t do work, and a philosophy does not meet goals - people do. I encourage you to seek out ways of working better with your fellow people.

Gratitude

I’d like to thank my friends for listening to me rant, and my editor Shaun Mouton for their help bringing this article together. I’d also like to thank the SysAdvent team for putting in the effort that keeps this fun tradition going.

Contact Me

If you wish to discuss with me further, please feel free to reach out to me. I am gwaldo on Twitter and Gmail/Hangouts, and seldom refuse hugs (or offers of beverage and company) at conferences. Death Threats and unpleasantness beyond the realm of constructive Criticism may be sent to:

Waldo
c/o FBI Headquarters 
935 Pennsylvania Avenue, NW
Washington, D.C.
20535-0001

Day 16 - How to Interview Systems Administrators

Written by: Corey Quinn (@quinnypig)
Edited by: Justin Garrison (@rothgar)

There are many blog posts, articles, and even books[0] written on how to effectively interview software engineers. Hiring systems administrators[1] is a bit more prickly of a topic, for a few reasons.

• You generally hire fewer of them than you do developers[2].
• A systems administrator likely has root in production. Mistakes will show more readily, and in many environments “peer review” is an aspiration rather than the current state of things.
• It’s extremely easy to let your systems administration team become “the department of no.” This can have an echo effect that pumps toxicity into your organization. It’s important to hire someone who isn’t going to add overwhelming negativity.

Every job interview since the beginning of time is built around asking candidates three questions. They’ll take different forms, and you’ll dress them up differently each time, but they can be distilled down as follows.

1. Can you do the job?
2. Will you like doing the job?
3. Can we stand working with you?

Doing the Job

This is where the barrage of technical questions comes in. Be careful when selecting what technical areas you want to cover, and how you cover them. Going into stupendous depth on SAN management when you don’t have centralized storage at all is something of a waste of time.

Additionally, many shops equate trivia with mastery of a subject. “Which format specifier to date(1) will spit out the seconds since the Unix epoch began?” The correct answer is of course “man date” unless they, for some reason, have %s memorized– but what does a right answer really tell you past a single bit of data? Being able to successfully memorize trivia doesn’t really speak to someone’s ability to successfully perform in an operational role.

Instead, it probably makes more sense for you to ask open ended questions about things you care about. “So, we have a lot of web servers here. What’s your experience with managing them? What other technology have you worked with in conjunction with serving data over http/https?” This gleans a lot more data than asking trivia questions about configuring virtual hosts in Apache’s httpd. Be aware that some folks will try to talk around the question; politely returning to specific scenarios can help refocus them.

Liking the Job

Hiring people, training them, and the rest of the onboarding process are expensive. Having to replace someone who left due to poor fit, a skills mismatch, or other reasons two months into the job is awful. It’s important to suss out whether or not the candidate is likely to enjoy their work. That said, it’s sometimes difficult to ascertain whether or not the candidate is just telling you what you want to hear. To that end, ask the candidate for specific stories regarding their current and past work. “Tell me about a time you had to deal with a difficult situation.” Push for specific details– you don’t want to hear “the right answer,” you want to know what actually happened.

This questioning technique leads well into the third question…

Not Being a Jerk

If you think back across your career, you can probably think of a systems administrator you’ve met who could easily be named Surly McBastard. You really, really, really don’t want to hire that person. It’s very easy for the sysadmin group to gain the reputation as “the department of no” just due to their job function alone– remember, their goal is stability above all else. Your engineering group (presuming a separate and distinct team from the operations group) is trying to roll new features out. This gives way to a natural tension in most organizations. There’s no need to exacerbate this by hiring someone who’s difficult to work with.

A key indicator here is fanaticism. We all have our favorite pet technologies, but most of us are able to put personal preferences aside in favor of the prevailing consensus. A subset of technologists are unable to do this. “You use Redis? Why?! It’s a steaming pile of crap!” is a great example of what you might not want to hear. A better way for a candidate to frame this sentiment might be “Oh, you’re a Redis shop? That’s interesting– I’ve run into some challenges with it in the past. I’d be very curious to hear how you’ve overcome some challenges…”

Remember, the successful candidate is going to have to deal with other groups of people, and that’s a very challenging thing to interview for. It also helps to remember that interviewing is an inexact science, and everyone approaches it with a number of biases.

For this reason, I strongly recommend having multiple interviewers speak to each candidate, and then compare notes afterwards. It’s entirely possible that one person will pick up on a red flag that others will miss.

Ultimately, interviewing is a challenge on both sides of the table. The best way to improve is to practice– take notes on what works, what doesn’t, and adjust accordingly. Remember that every hire you make shifts your team; ideally you want that to be trending upwards with each successive hire.

[0] I’m partial to http://smile.amazon.com/Recruit-Hire-Great-Software-Engineers/dp/143024917X?sa-no-redirect=1 myself.
[1] For purposes of this article, “systems administrators” can be expanded to include operations engineers, devops unicorns, network engineers, database wizards, storage gurus, infrastructure perverts, NOC technicians, and other similar roles.
[2] For purposes of this article, “developers” can be expanded to include… you get the idea.

Day 9 - D3 for SysAdmins

Written by: Anthony Elizondo (@complex)
Edited by: Shaun Mouton (@sdmouton)

In this post we will talk an introductory look at D3.js and using it on relatively "raw" datasets such as those in CSV or JSON.

D3.js is a powerful Javascript library that allows you to build SVG images and animations, implemented with HTML5 and CSS.

There are some who will tell you D3.js has a steep learning curve. I think the reason for this is it lives at the center of a Venn diagram where the circles are Javascript, CSS, math, and graphic design. Not the most common skill set.

But don’t let that scare you. You’re awesome! You can do it!

Visualizing your Data

Are you collecting metrics? A recent survey by James Turnbull of Kickstarter says that 90% of you are. (If not, why not?) Ideally these metrics are fed into a permanent installation of a tool that can be used for measuring performance, alarming on errors, or just general trending. And that is great.

But perhaps you just want to whip up something on an ad-hoc basis. D3.js is superb at this. With simply a CSV file and a web server (even something as simple as "python -m SimpleHTTPServer 8000", see William Bowers' list) you can create something that works on any modern browser, mobile included.

Data Type and Format

Before we can do anything, we have to think about what type of data we have. More precisely, what is the story we want to tell about it? We might have a collection of ohai output (JSON) from 10,000 servers. This type of data would be nicely visualized with a force-directed graph, perhaps even sorted by type. Maybe you have logs of concurrent sessions on HA Proxy or your F5 load balancer. Then this heat map would work well. Your data might be as simple as a single metric, but you want to chart it over time. A simple line chart would be sufficient.

Fundamentals of D3.js

D3.js operates on the DOM of a web page. You can boil most of its operation in three phases. First, it creates SVGs and adds ("appends") them to the page. Next it reads in data embedded in the page itself, from a separate file, or from other online source via AJAX. Then, it performs transforms to the SVG elements based on this data, or perhaps based on user input.

Simple Example

Now we’ll try create a simple example. Assume our data, after some awk mangling, is in a TSV. The first column is the date in YYYY-MM-DD format, the second is a scalar indicating how many servers we have.

date   servers
2013-12-08  4343
2013-12-07  4328
2013-12-06  4325

First, include D3 in our HTML file,

<script src="http://d3js.org/d3.vs.js"></script>

Now create the SVG.

var svg = d3.select("body").append("svg")
.attr("width", 860)
.attr("height", 500)
.append("g")
.attr("transform", "translate(" + margin.left + "," + margin.top + ")");

Time to load our data.

d3.tsv("vmcount.tsv", function(error, data) {
data.forEach(function(d) {
d.date = parseDate(d.date);
d.servers = +d.servers;
});

And draw it.

svg.append("path")
.datum(data)
.attr("class", "line")
.attr("d", line);

The .append("g") and attr("d", line) are not magic. They are SVG element tags. The "g" attribute indicates "group everything together as one". The "d" attribute defines a path to follow. The 3 bits of Javascript above are the core of the work, but they alone are insufficient. There is a bit more required to bring it to life and make it look nice. That includes defining axes, scales, labels and domains (in the geometric sense). The full working example can be found here. It adds all the features mentioned above, plus a fancy hover to show precise values.

On Your Way

Hopefully I’ve demystified D3.js for you a bit with this short introduction. To dive deeper I suggest browsing some of Mike Bostock’s simpler examples, and don’t be afraid to ask your friendly neighborhood frontend developer for help! :) The full D3.js API can be found here.

Resources:

• Interactive Data Visualization for the Web by Scott Murray (@alignedleft).
• D3.js Data Visualization Fundamentals by Ben Sullins (@bensullins).

Day 8 - From Operator to Guide: Lessons Learned Moving from Engineering into Management

Written by: Mathias Meyer (@roidrage)
Edited by: Ben Cotton (@funnelfiasco)

As a company grows from a tiny startup to a small business, curious things happen. The small startup struggles to figure out how to scale a product and the infrastructure supporting it, with as few people as possible in as many roles as possible.

The small business struggles with other pains like growing a team, creating a set of shared values, growing the customer base, and focusing on longer-term strategies. It’s not so much about survival anymore as it is about building something sustainable.

Our little company stepped from one phase into the other this year, and the challenges have been interesting, especially for someone like me, who went from being an infrastructure engineer to being a manager.

Our little company, an open source project at heart, has a history of trying to be as open as we can about our production issues and nurturing a blameless culture.

When you’re a small team, that can be an easy seed to plant and to get everyone on board. After all, everyone’s already responsible for everything, so switching hats is easy.

But as a manager, you’re now confronted with making sure your team and your customers are happy and that the business is growing (hopefully because your people and customers are happy).

That’s where push comes to shove with a blameless and just culture, balancing production pressure, people happiness and productivity into something that fosters critical thinking, a culture of always looking out for and learning from unexpected events, and making sure that your product’s stability is improved based on experience in production.

Letting go and delegating responsibility

As an infrastructure engineer, I’ve been involved in responding to a lot of our outages in our early days and years, writing the postmortems and giving public talks about our culture.

I still rotate through on-call and respond to production incidents, but our primary focus, as a team, has become to make sure that knowledge of handling outages is spread to as many people as possible. That meant one thing I’ve been having trouble doing over the past couple of months: letting go.

When you want other people to take on the responsibility to handle production incidents, you need to step back a little. You turn from being an operator to being a guide. You get out of the way.

The only way to build trust in your team is to let people learn from mistakes, whoever’s mistakes they are. As Richard Cook said in “How Complex Systems Fail”:

Failure free operations require experience with failure.

When you want your team to be able to respond to outages, make their own experiences and share them with the team, it’s important to let go. You can still be of a helping hand of course, you can ask (helping) questions during outages to get as much information as possible or to nudge people in different directions.

After all, everyone is prone to their own biases, and not everyone on your team will have experience will all aspects of your systems.

What did I do over the last couple of months? I’ve been a bystander for maintenances and outages, giving encouragement and getting stuff out of the way where I can.

People need to feel a sense of ownership and responsibility. If you as a manager continue to be the leading hand during outages, you’ll always be the first line of response and you’ll take away those experiences from your team.

Just a little bit of process

Handling a production incident is one part of the story. The other is what comes afterwards. You can choose to ignore what happened, but that’s leaving a great opportunity to learn on the table.

Depending on where your team members come from, this can be a challenge. Not every company fosters a culture of looking at what happens in production and to make sure the appropriate learnings are taken away and used for future improvements. Some prefer to find a root cause (usually a human) and fire them. Problem solved.

Others only have little experience in conducting both incident response and a postmortem. Two things are important to get everyone up to speed:

Mentoring

Teach your team what you know, share the knowledge of what’s considered a good incident response and the minimum requirements are for a postmortem.

Get those things into writing (an Ops Playbook is a good idea to have) and make sure that people read it, commit to it, and are actively involved in improving it.

Mentoring is one of your bigger tasks as a manager, and if you want to get people to take responsibility and to follow in yours or make their own footsteps, it’s the best place to start. You can mentor them personally or share knowledge internally by giving talks, feedback on previous incidents and postmortems, writing on your internal blog and creating discussions around what people think.

Setting Expectations

Once you’ve shared knowledge, you can agree on expectations. Those can also be manifested in the Ops Playbook.

In our case, internal postmortems need to be filed within 24 hours after the incident.

The idea isn’t to put high pressure on people, but to make up for the fact that humans quickly forget important details, in particular after a stressful situation.

If there was extended impact for our customers, a public postmortem should go up on our status page within 72 hours after the outage.

You may not need a strict process around this, but having a few guidelines helps encourage everyone to stop and think for a moment about what you can take away from an incident and how you can improve operations, development or your product in the future.

Production pressure and continuous improvement

As a manager, it’s too easy to fall into the trap of production pressure, especially when you’re reporting to other managers. You may be more accountable to stake- or share holders.

How can you combine this with the idea of continuous improvement, in particular if you have a lot of outages that require fixes and improvements?

The answer to that is surprisingly simple. When you don’t apply learnings from incidents, they will continue to interrupt your customers in whatever task your product is assisting them with, leading to increased unhappiness, churn and effectively reduced revenue.

One of our struggles recently has been to keep up with remediation items from previous outages. This is my fault, not my team’s. It’s a manager’s responsibility to give room for scheduling in the remediations their team deems necessary. It’s their responsibility to encourage everyone to be involved in the process.

When remediation items aren’t cleared up, it’s the manager’s responsibility to find out what’s blocking the team and to get that out of the way.

I encourage you to look at how Etsy is conducting postmortems, how they store them and making sure that remediation items are followed up on.

Nudging people to think critically

Traditional thinking (or the Old View of Human Error (see “The Field Guide to Understanding Human Error”, Sidney Dekker) suggest that there is a root cause for any incident. The Five Whys suggest that it only takes a few questions to find it.

Root cause is appealing to engineers because it gives them one thing to fix. Root cause is appealing to management because it gives them one thing to blame (and to fire).

Root cause analysis avoids critical thinking. When your team stops at a single point and focuses only on that, you remove any opportunity to figure out the bigger picture, the motivations people had at the time of the incident or during events leading up to them.

Thinking beyond the root cause requires critical thinking and asking questions, as uncomfortable as they may be. Just so we’re clear, they can and should be uncomfortable to you as a manager as well.

One of our challenges has been encouraging people to think beyond a simple resolution to an outage, beyond just this one cause.

Sometimes that means asking them a few more questions, sometimes you as their manager need to ask yourself questions on how you can improve the organizational environment or the processes that people were acting based on.

From generalization to specialization

In the early days of most startups, you’ll find everyone doing everything. We were firm believers in generalization in the early days, as both our team size and the little revenue didn’t allow for specialization early on.

Today we find ourselves working towards more specialized roles. We encourage people to grow into more specific roles or responsibilities (without stressing on the titles too much) to increase the level of ownership people feel about our product, but also to remove ambiguity in who’s responsible for what.

We started longing for specialization, not just in engineering, as our business grew to a stable customer base and revenue.

That includes defining responsibilities in incident response. Who’s the main point of communication? Who facilitates the team’s response? How many people should be involved in a scheduled maintenance? How and when are issues escalated to involve more people?

All those questions can add a little bit more process, but they help in reducing ambiguity. Because when everyone’s responsible for everything, is anyone really responsible?

Towards a culture of learning

Running a business with customers and a team distributed across the globe makes it hard to stop and reflect on what you’re doing, on the things that have happened in production and to figure out what you can learn from it.

It’s easy to just keep going and leave things on the sidelines, thinking “We can always fix this later.”

But whatever you leave on the sidelines is going to come back and bite you later, causing more embarrassment when it affects your customers.

Continuous learning requires everyone to take a step back, and it requires you as a manager to give everyone room to do so, room to improve, room to learn and to gain more experience.

Be the manager you always wanted to have

Growing into a manager role is a unique chance to be the manager you always wanted to have. You’ll have to incorporate a bigger picture into this idea, but it’s up to you to shape how you treat your team and your production system. That doesn’t mean leaving aside everything your previous managers did, because some of that may have a good reason.

You can build a culture based on the old view of human error (believe me, it’s all too easy to fall into this trap), or you can treat your team with the empathy that you always wanted from your previous managers (yes, those who always put the schedule first and people last).

The important thing to remember is that as a manager, you need to trust the people at the sharp end of the action to do the right thing. It’s very likely the very same thing you would’ve wanted your manager to do or that you expected as an operator.

As Richard Cook said:

Actions at the sharp end resolve all ambiguity.

Your role as a manager is now to reduce ambiguity over time. If you don’t, your engineers will do it for you, giving you plenty of opportunity to learn and improve their environment, possibly causing unhappiness all around.

It’s been an interesting change in role for me, and I’m bound to make mistakes and learn from them. But learning is what this is all about, for you, your team, and your company.

If you have any stories to share or want to talk about culture and management, I’d love to talk. Shoot me an email.

Day 7 - What System Administrators Can Learn From Community Management

Written by: Daniel “phrawzty” Maher (@phrawzty) & Jonathan Clarke (@jooooooon42)
Edited by: Yakira Dixon (@yakiradixon)

Introduction

At first blush it might not seem like there are any parallels between system administration and community management, but the two fields have much more overlap than you might think! Before we begin, let’s define community management briefly, and to do so, I’m going to lean on this this excellent post by Deb Ng. It’s not marketing, nor social media, nor tech support, though these things can be involved. A community manager is a strategist, a content creator, and above all, a communicator. In this article, we will explore a handful of key principles from the community management world and see how they apply to system administration.

It’s important to realise that both fields are concerned with systems, which is to say, “an assemblage or combination of things or parts forming a complex or unitary whole” (source). In the case of sysadmins, one normally thinks of the relationship between nodes in a cluster, or clusters in a farm, or farms in a data centre, or - well, you get the idea. For community managers, one normally thinks of the relationship between members of a forum, or forums in a site, or sites in a community, or - well, here again, you get the idea. In both cases, the relationships between the individual elements of the whole can be complex, and often require specialised tools and years of experience to get a handle on.

Managing groups at scale

Scale is an important concept in community management. What works for managing a group of two people might not work for a group of 20 or 200 people. It’s generally easier to discuss a topic in a small group and achieve consensus when there are mutually-shared interests. Small groups can function directly together since the number of individual voices is low and the space needed for them to work is commensurate with their size. Logistically speaking, a small group can make use of pretty much any communication platform that happens to be convenient - a common list of people to CC when emailing and a Google document might well be enough.

Contrast that with managing a community of, say, 50 people - that’s a big list of people to keep tabs on every time you need to write an email (not to mention the amount of noise that might get generated with an errant reply- all), and as anybody who’s ever tried to play a fast-paced game of multiplayer text editing knows, that’s a lot of cooks in the kitchen when it comes to a shared document. Even at this modest scale, some sort of more purpose-built tooling is required, such as a proper mailing list or forum site. This necessity is borne from two important effects of larger groups which, for our purposes, we can refer to as quantity and quality.

Quantity is straightforward: more people means more conversations! A handful of emails per week is trivial for any one person to deal with, but multiply that by a factor of ten or 100, and the trickle becomes a raging torrent. The more information that’s being generated, the more difficult it becomes for any one member of the group to process it effectively.

Quality refers to the nature of those conversations, and in particular, their level of diversity. Each person in the community will bring their own particular histories, viewpoints, cultural perspectives, and so forth to the group. Good tooling can make all the difference when trying to manage such a heterogenous system. A straightforward example of this is mature forum software that has conversational threads, thematic areas, and a robust back- end for gathering statistics and metrics about usage.

From here it shouldn’t be difficult to make the mental leap to system administration. Two servers sitting on your desk are pretty straightforward to deal with - even if they’re totally different, there’s only two of them, so you probably don’t need any special infrastructure to manage them. Scale that up to a dozen servers, however, and the situation becomes much more complex. The dual effects of quantity and quality are instantly applicable. At even relatively small scales the importance of a coherent configuration management tool becomes evident: something that allows the different nodes in the system to be organised as operational or thematic groups, and with a robust back-end for gathering statistics and metrics about usage. Sound familiar?

We’re going to come back to scale later, but for now, let’s move on to something a little bit more esoteric.

Writing a mission statement

I know what you’re thinking: “mission statement? That’s just corporate-speak nonsense!” And yes, that can certainly be the case, but it doesn’t have to be. In fact, a good mission statement is essential for any community, and explicitly stating it can help to crystallise the community in powerful ways.

Most organic communities assemble around an initial idea or principle. While this shared element is enough to form the community, it’s rarely enough to maintain it, and almost certainly insufficient for real growth - the secret sustainability sauce is made up of the values that original germ represents. In other words, people come for the brilliant spark, but they stay for the ideas generated from the spark. A mission statement is a way to codify and present both.

Sound a little high level? Consider that at some point in your life you’ve probably come across an open source project that was little more than a few hundred lines of code and a README. But something about this project made it different from the others. The documentation, ersatz though it might have been, clearly elucidated the problem and how this project acted as a (potential) solution. This simple addition to the repository made all the difference in the world because it helped you to understand how the project could fit your particular situation. Even without a deeper analysis of the code in question, your interest was piqued and your desire to learn more was set into motion. What happened there? Believe it or not, what captured your attention was, fundamentally, a mission statement.

Most established companies have mission statements. These are organisation- wide screeds that may or may not have any direct relationship with the work that any one contributor effects within that organisation. Is it possible to bring the granularity level down a notch or two, and if so, what might the benefit be? Could it be useful to a system administration team?

Imagine you’re just starting on a new project: a simple but powerful collection of scripts to manage virtual machines that’s going to make everyone’s day-to-day work much easier. Right now, even before you’ve begun writing any code, is the perfect time to write a mission statement. Start with something simple like “I’m fed up of typing in a dozen long and complicated commands every time I need to migrate a VM in our infrastructure, especially in the middle of the night, when on call. I intend this to be a small wrapper tool that can do normal migrations for us by typing in just one command. This should reduce the amount of time we waste on manual migrations and help to eliminate typing errors”. This brief, simple statement encapsulates both the problem and the solution, clearly defines the scope of the project, and describes both the initial idea germ and - critically - the parameters for success.

There are numerous benefits to drafting your mission statement at the earliest possible stage. Crucially, by acting as a sort of combination compass and mold, it acts as a guide to keep the contributors focused (even if you’re the only one), which helps to nip things like feature creep in the bud straight away. Keep it short and don’t overthink it. Much like setting up some new test infrastructure, or working on a new program, the best thing is to start small and iterate over time as necessary. Finally, writing a mission statement should be easy; if you’re embarking on a project, then you know best what problem you’re trying to solve, and at those early stages is when you’ll have the most clarity about it. If for no one else, write the mission statement for your future self - a few weeks down the line you may have gone so deep you completely forgot the original problem you were trying to solve!

A good mission statement can also be used to generate excitement which, as we’ll see below, can be a powerful tool for getting things done.

Encouraging involvement

A key principle of community management is actively encouraging contribution and involvement from the community. A good community manager will generate excitement and foster enthusiasm. Cultivating a positive emotional connection to the community will make members more loyal and encourage them to participate even more - a virtuous cycle that in turn makes the community even stronger. That said, while a general feeling of excitement is, like that initial idea germ, enough to create a spark, it’s insufficient to maintain the community. The excitement needs to be backed up by concrete measures to encourage active involvement.

There is a fairly well-known philosophy in the open source world called “Release early, release often” that was popularised by Eric S. Raymond in the late 90’s. The essence of the philosophy is that sharing versions of a software project early and often will ultimately improve the overall quality of the software. This happens because beta-testers and early adopters can have fun playing with new features while contributors can learn the codebase by fixing obvious bugs - that early feedback will help influence development in useful ways. In fact, as long as they’re not fatal, it can be helpful to leave some bugs in there deliberately! Providing low hanging fruit can enable an enthusiastic contributor to jump into the project by giving them the chance to fix that bug themselves.

Looking at this from a systems perspective, this is basically a process with a feedback loop, and by releasing often the loop is accelerated. In other words, excitement is generated by rapid development, and the rapid development is encouraged by excitement - it’s almost too easy!

As a system administrator, while you may certainly be leading up a project for which the rapid release model is applicable, that won’t always be the case. That’s no reason not to adopt this principle, however; by actively encouraging your co-workers to get involved, you help to build a trust relationship that will pay dividends down the line. Consider the development vs. operations silo for example - how nice would it be if the two groups actually worked together on projects right from the start?

So how do you generate excitement? The easiest thing is to talk to your co- workers. Write a mission statement - or, better yet, draft one in collaboration with some potentially interested parties. Ask questions, discuss ideas, and continuously act on that feedback. Share design documents, proofs of concept, and initial implementations. Don’t be shy about unfinished releases or weird bugs, and don’t let perfection be the enemy of good - the important thing here is that both the features and the bugs are clearly explained.

Whatever the scope of your next project, try sharing the idea with your co- workers (or whomever) before you get started and see how they react. Chances are, you’ll get their support a lot more easily that by showing them hundreds of lines of script that don’t yet solve the whole problem - and they probably won’t nitpick about your chosen scripting language or variable naming while reading the mission statement either.

Of course for others to be able to fix bugs in your code, they need to know how to get that bug fix back into the code. Which brings us to our next topic"¦

Establishing community guidelines

A simple search with the term “community guidelines” will reveal hundreds of thousands of examples from across the web. They encompass, either implicitly or explicitly, communities of just about every conceivable size on more or less every topic there is. It’s difficult to find a legitimate site that doesn’t have one - but, what are they? Simply stated, they are the basic tenets that frame the social contract between all of the members of the community, and they serve as ground rules to help that community to interact well. For example, an increasingly popular type of guideline is the conference code-of-conduct, which most conferences have now to help ensure that respectful and inclusive behaviour prevails.

Responsible community management is, in many respects, predicated on the existence of community guidelines. It’s one of the most important tools that leaders can use to act decisively and consistently.

When technical folk collaborate, be it in writing code or in administrating a server, we all tend to do things slightly differently from one another. How so? A simple question: where do you put custom binaries on a server?

Right.

Now, chances are you’re thinking “I don’t need to write out guidelines - we’re a tiny group”. You may well be right - your group probably can function just fine, for now; however, consider when a new person is joining your team. They will need to understand “how to do stuff around here”. Thinking further, the future could include a larger expansion, perhaps open sourcing a tool you wrote, or your company being acquired or merging with a larger one. In these cases, having planned ahead and written down the key points on how to work together could be a life saver - for both yourself and the new arrivals.

As with everything, the key is to start simply: put a minimum viable set of guidelines together and then iterate as necessary. Some good starting points might include things like “all code must live in a given version control system.” That’s a simple, effective guideline that is both easy to follow and hard to misinterpret. Stipulate the conventions up front: paths on file systems, tabs vs. spaces for indentation, and so forth. That’s a great start.

Next, explain how to contribute; this could be by sending a pull request on GitHub, a patch by email, or whatever. The important thing is to set up a consistent framework. Elucidate expectations for responses to contributions, too - it’s better to say something like “I only review patches on Mondays, so please be patient” than to leave an enthusiastic first time contributor hanging for six days. Examples of such standards documents (that have open source licenses) include Google’s style guide, or for a more simple example, Rudder’s technical guidelines.

These contribution guidelines obviously tie into lowering the barrier to entry to your project too (see previous section). The underlying goal is to facilitate collaboration by avoiding pointless debates about which indentation style to use, when the project could be accepting contributions and moving forwards. There are no hard and fast rules governing the content or structure here, but as a general rule the idea is to be concise - use clear, brief language that expresses the core principles accurately.

Similar guidelines can, and should, be drawn out for social interaction. Don’t fall into the trap of defining complex rules that are hard to follow and, most importantly, assume good intentions by default. Great relationships are built on trust, and the best way to get trust is to give it in the first place. There is really no one-size-fits-all set of rules and each community will grow around the shared values that have particular meaning to them; however, it is very important to point out what is not OK. Again, exact details here depend on who you are as a community but make sure that as a bare minimum you explicitly forbid any kind of discriminatory behaviour or harassment.

Last but not least, remember that as one of your community’s leaders - however big or small that community is - you should be setting the example through action. Respect your own conventions. Welcome newcomers. Communicate clearly, consistently, and often. Be mindful that your words carry weight, and choose them deliberately and carefully when you write every email, blog post, or README.

Resources

The web is full of resources about community management - if you’re curious to learn more, some good starting points (for system administrators) include the Producing Open Source Software book, Eric S. Raymond’s The Cathedral and The Bazaar, and The Mozilla Manifesto. Dawn Foster’s blog also has some great insight on managing a community from a sysadmin perspective.