Day 9 - 3 things parenting taught me about system administration

By: Jennifer Davis (@sigje)

The last five years have been grounding for me as I became a beginner at parenting. In this article, I want to share three things I learned about being a better sysadmin from being a mom.

Prioritize your health

Of course, I've heard it so many times. But in the rush of trying to support the "system," sometimes, I lose track of the little things (getting enough sleep, eating meals, human engagement that isn't predicated on deliverables and action items). When it comes to parenting, I see the difference in how the necessities of the moment can gradually subsume the primary goals and real joy* (a secondary outcome of successful parenting that I tend to only enjoy in retrospect, after having assured myself that my internal parenting kanban board is as it should be–obsession, exhaustion, and then joy tends to be my experiential flow as a parent).

Prioritizing health - if I'm not ok, I'm not able to handle the "system" as well, regardless of its state.

Any parent of a child under five will tell you that 90 percent of the job is keeping the child alive. If they make it to the next day, smile and giggle the proper number of times per day, and if your friends, family, and parenting peers seem unaware that your parenting path bears a concerning resemblance to the plot of the movie Speed, then you're more or less gravy. You also learn that, while you can spend a great deal of time analyzing and conversing about your child and how they're faring, the main thing is to put them in the right places at the right time. Sunshine, exercise, the company of their peers, easily accessible bathrooms–these are the things that matter. If my son doesn't get direct sunlight within 90 minutes of walking, his mood takes a nosedive, and this isn't a mystery to me. Likewise, if he isn't let loose at the park to terrify small woodland creatures with his desire to befriend them, his attentional resources will be suboptimal when it's time for flashcards. Yet I (and I don't think I'm alone in this) will frequently wake, obtain caffeine, have a quick all-hands with my family, and proceed to sit in a small room staring at a screen for eight hours straight. As a result, my ability to practice self-care myself fails regularly.

Leverage the community

To prioritize my health, I have to ask for help. I've had the following experience again and again professionally, and as a parent, and at some point, I hope that it won't astound me, which it does every time: I believe that I'm having a singular experience (which, of course, we all are) and that I am an outlier because obviously no one else is concerned about the state of affairs or struggling. And then someone else gives voice to the precise issue that I've devoted considerable resources to NOT sharing. Of course, other people are also concerned about the children pretending that the scissors are boomerangs. One of my primary errors is thinking that there is some scorekeeping of tracking the social currency and categorizing discourse into the buckets of "I helped" and "I was helped." It's a binary that renders engagements as transactional when my actual community experience is almost always that I walk away feeling better regardless of who broached a topic.

You can't eliminate all Snowflakes

Within the community, we often talk about snowflakes as problems. Yet, as a parent, you discover that there are no handbooks for YOUR kid because every child is different in their own beautiful, hard, and surprising way. Likewise, while there is value in the community and sharing stories, every system will be different. You work with one system, you've learned about that system, and while there are useful things you'll learn from that system to apply to other systems, every system will be beautiful, different, and hard in its surprising ways.

Wrapping Up

Our industry is constantly evolving with the introduction of new technology, tools, and processes. It may feel overwhelming to try to understand everything. You have to accept some degree of the unknown. When I first became a parent, I realized that Operations had prepared me for the inevitable changes that occur every single day. No matter what tomorrow brings, the essential skills are learning to adapt to change and learning to learn fast.

Please make time for yourself, connecting with the community, and accepting what is different and unique about your systems and the environments they are running in.

Day 6 - No More On-Call Martyrs

Written By: Alice Goldfuss (@alicegoldfuss)
Edited By: Justin Garrison (@rothgar)

Ops and on-call go together like peanut butter and jelly. It folds into the batter of our personalities and gives it that signature crunch. It’s the gallows from which our humor hangs.

Taking up the pager is an ops rite-of-passage, a sign that you are needed and competent. Being on-call is important because it entrusts you with critical systems. Being on-call is the best way to ensure your infrastructure maintains its integrity and stability.

Except that’s bullshit.

The best way to ensure the stability and safety of your systems is to make them self-healing. Machines are best cared for by other machines, and humans are only human. Why waste time with a late night page and the fumblings of a sleep-deprived person when a failure could be corrected automatically? Why make a human push a button when a machine could do it instead?

If a company was truly invested in the integrity of its systems, it would build simple, scalable ones that could be shepherded by such automation. Simple systems are key, because they reduce the possible failure vectors you need to automate against. You can’t slap self-healing scripts onto a spaghetti architecture and expect them to work. The more complex your systems become, the more you need a human to look at the bigger picture and make decisions. Hooking up restart and failover scripts might save yourself some sleepless nights, but it wouldn’t guard against them entirely.

That being said, I’m not aware of any company with such an ideal architecture. So, if not self-healing systems, why not shorter on-call rotations? Or more people on-call at once? After all, going 17 hours without sleep can be equivalent to a blood alcohol concentration of 0.05%, and interrupted sleep causes a marked decline in positive mood. Why trust a single impaired person with the integrity of your system? And why make them responsible for it a week at a time?

Because system integrity is only important when it impacts the bottom line. If a single engineer works herself half-to-death but keeps the lights on, everything is fine.

And from this void, a decades-old culture has arisen.

There is a cult of masochism around on-call, a pride in the pain and of conquering the rotating gauntlet. These martyrs are mostly found in ops teams, who spend sleepless nights patching deploys and rebuilding arrays. It’s expected and almost heralded. Every on-call sysadmin has war stories to share. Calling them war stories is part of the pride.

This is the language of the disenfranchised. This is the reaction of the unappreciated.

On-call is glorified when it’s all you’re allowed to have. And, historically, ops folk are allowed to have very little. Developers are empowered to create and build, while ops engineers are only allowed to maintain and patch. Developers are expected to be smart; ops engineers are expected to be strong.

No wonder so many ops organizations identify with military institutions and use phrases such as “firefighting” to describe their daily grind. No wonder they craft coats of arms and patches and nod to each other with tales of horrendous outages. We redefine what it means to be a hero and we revel in our brave deeds.

But, at what cost? Not only do we miss out on life events and much-needed sleep, but we also miss out on career progression. Classic sysadmin tasks are swiftly being automated away, and if you’re only allowed to fix what’s broken, you’ll never get out of that hole. Furthermore, you’ll burn out by bashing yourself against rocks that will never move. No job is worth that.

There is only one real benefit to being on-call: you learn a lot about your systems by watching them break. But if you’re only learning, never building, you and your systems will stagnate.

Consider the following:

1. When you get paged, is it a new problem? Do you learn something, or is it the same issue with the same solution you’ve seen half a dozen times?
2. When you tell coworkers you were paged last night, how do they react? Do they laugh and share stories, or are they concerned?
3. When you tell your manager your on-call shift has been rough, do they try to rotate someone else in? Do they make you feel guilty?
4. Is your manager on-call too? Do they cover shifts over holidays or offer to take an override? Do they understand your burden?

It’s possible you’re working in a toxic on-call culture, one that you glorify because it’s what you know. But it doesn’t have to be this way. Gilded self-healing systems aside, there are healthier ways to approach on-call rotations:

1. Improve your monitoring and alerting. Only get paged for actionable things in an intelligent way. The Art of Monitoring is a good place to start.
2. Have rules in place regarding alert fatigue. The Google SRE book considers more than two pages per 12 hour shift too many.
3. Make sure you’re compensated for on-call work, either financially or with time-off, and make sure that’s publicly supported by management.
4. Put your developers on-call. You’ll be surprised what stops breaking.

For those of you who read these steps and think, “that’s impossible,” I have one piece of advice: get another job. You are not appreciated where you are and you can do much better.

On-call may be a necessary evil, but it shouldn’t be your whole life. In the age of cloud platforms and infrastructure as code, your worth should be much more than editing Puppet manifests. Ops engineers are intelligent, scrappy, and capable of building great things. You should be allowed to take pride in making yourself and your systems better, and not just stomping out yet another fire.

Work on introducing healthier on-call processes in your company, so you can focus on developing your career and enjoying your life.

In the meantime, there is support for weathering rough rotations. I started a hashtag called #oncallselfie to share the ridiculous circumstances I’m in when paged. There’s also The On-Call Handbook as a primer for on-call shifts and a way to share what you’ve learned along the way. And if you’re burned out, I suggest this article as a good first step toward getting back on your feet.

You’re not alone and you don’t have to be a martyr. Be a real hero and let the pager rest.