Day 1 - Comfortably Numb - or - “How Corporate Analgesia is Actively Harming You”

Written by H. “Waldo” Grunenwald, (@gwaldo) Edited by Adam Compton (@comptona)

Hi, I’m Waldo, and this is “Comfortably Numb: How Corporate Analgesia is Actively Harming You.” In short, I’m here to talk about pain.

Congenital Analgesia

Congenital analgesia is a medical condition whose sufferers are completely unable to feel physical pain. At first thought you might think it is a superpower…

Unfortunately, congenital analgesia is not a superpower. It doesn’t grant you better reflexes, or make you super-nimble, or prevent you from bleeding out. In fact, you’re probably going to be clumsy, because our spatial awareness is built mostly from our sense of touch.

As a result of being clumsy, you have to be extra vigilant about your surroundings, and your self. People with congenital analgesia have to frequently check themselves for cuts, bruises, and broken bones, in order to avoid or treat infections they haven’t noticed because it doesn’t hurt. Additionally, people suffering from it may also experience weakness, in that the unresponsive nerves cannot accurately control the muscles that they’re associated with. We’ll come back to this.

Do you work at a company where the org chart looks something like this?

Do you have a group responsible for making things, and another group responsible for keeping those things available? If so, I’ll bet that new features seem to take a long time to make available once they’re written. Do you have problems with handoffs? Does it take a long time to get things resolved when multiple groups need to get involved? Not that there’s fingerpointing…

But it’s funny how everyone’s theory of what’s broken involves someone else doing work…

… Huh…

I’m going to talk about something that I normally wouldn’t bring up in conversation, and certainly not in a presentation or article.

I’m going to do this because I need you to feel something. Now, I have a confession:

Confession

(This section includes descriptions of pain and mild violence; if you prefer not to read that, please click here to skip ahead.)

After elementary school, I played baseball for a few years. I gave it up mid-high-school, because I wasn’t great, and I wasn’t a coach’s kid. So, please take me at my word when I say that I was a decent catcher.

For those who don’t know: in baseball, the pitcher’s (person in the middle of the diamond) job is to throw the ball past the batter, but close enough for the batter to have a chance to hit it. If the pitcher successfully throws the ball past the batter, it is caught by the catcher (the person behind home plate).

As a catcher, I had a lot of extra protective gear that the other players didn’t need, but there were two pieces of safety equipment that were more important than the rest:

The reasons for this should be obvious, but to be explicit: a baseball traveling 60 miles per hour hitting your face or gender-parts is incredibly painful.

This pain is not exclusive to men; many female catchers wear a cup, too.

Even if it’s not from baseball, most people have had some experience with being hit in the gender-parts. It’s a very special kind of pain, isn’t it?

I want you to remember the feeling. That particular all-encompassing, paralyzing, electrocuting feeling. Really savor it for a moment.

If you see someone get kicked in their gender-parts, you’re going to feel something, right? Sympathy pain. Not nearly as intense as being hurt yourself, but there’s still something there.

When someone tells you about being hurt, but you didn’t see it happen, you might feel a little sympathy pain, but it’s easier to shrug off. Now, let’s pretend that it’s time for a demonstration…

(No, not really.)

Let’s say I call two people up to a stage: Terry and Sam. If you see me kick Terry in the gender-parts, all of you in the audience are going to react to with sympathy pain for Terry. But Sam will be feeling something different…

In addition to the sympathy pain, Sam will be wondering what his role is going to be. Will Sam be doing the kicking this time, or will be he getting kicked? And since he wasn’t asked to do anything yet, Sam now expects to get kicked. Sam now has added the sinking feeling of “Oh no, I’m next.”

Sam’s anticipation has spiked. It’s possible his sympathy pain might even be overshadowed by the anxiety he’s now feeling too.

And this is why being on-call is exactly like being kicked in the gender-parts.

It’s not just the heart-skip feeling of being woken up at 3am.

It’s the frustration of the 5th page that came in, one on top of the next, when someone hit all of the deploy buttons even after notice was sent out that the artifact repository was down, and paged each button’s failure as a separate page.

It’s the feeling of being three yaks into a “why the hell is it doing this?!“ problem, but you lose all of that context that you’d built when the pager goes off.

(It was DNS, BTW.)

When I tell you that I got no sleep because I got paged all night, you can feel sympathy for me. Especially if you’re on the same rotation with me. If you’re not on an on-call rotation at all, it’s like being told about being kicked in the gender-parts instead of watching it happen: your response is probably going to be just “that sucks, sorry.”

Why?

Because if you’re not the primary on-call for your products, you’re Comfortably Numb. You simply don’t have to care as much.

Don’t get me wrong: nobody is calling you a bad person. Sure, you have a professional mandate to make quality products. However, if you’re not the primary on-call for your application, you don’t have skin in the game.

Hearing about an outage is a far cry from being awakened at 3am to diagnose and fix it. I’d argue that it’s an even further gulf than from seeing someone get kicked in their gender-parts than being kicked yourself.

If you’re not primary on-call for your product you can sympathize, but you’re safe in the knowledge that you’re not going to be kicked. This has the added effect of contributing to “Us Versus Them” thinking. Because the people who care for your products know that there is a hierarchy-of-status.

Yes, a hierarchy of status…

And that’s fine for the majority. Right up until you’re told that you’re now going to serve as primary on-call for your apps.

WHOA WHOA WHOA WHOA

WHAT HAPPENED THERE?!

Suddenly you went from a bystander to being next to the person who just got kicked!

It’s going to be ok.

Now, I’m here to tell you that it’s going to be ok.

Maybe you’ve never done this before. We know you’ll need help. You will have help. You will learn. You will grow.

You’ll have a greater appreciation for people who run things. Because now you, too, run things. You’ll understand that failing gracefully is an art, and requires consideration.

You’ll lose patience for “normal errors” in logs and for black-box software that doesn’t allow for introspection. You’ll develop a love for insight tooling, and develop a passion for creating products that emit their own status.

And, you will fail.

You will screw up horribly.

And it will be ok.

It will be ok because your company will realize that this is a tough transition, and that this is an investment in you and your products.

So you will fail, but you won’t be alone. You won’t be a failure. Here’s the Thing…

See, those people who used to Run your things…

Well, they’re still around!

Yes, it’s a bit on-the-nose for me to refer to my past career as superheroes… BUT the number of times that one of my peers has been MY HERO is amazing. Nobody knows everything! You just can’t.

You can only be an expert in so many things, but you can learn to be basically competent in so many more! Those Runners-of-Things will still be the experts in finding and preventing failure modes. They can and will teach you to competence in Running things.

I told you all of that so I could tell you this:

If you are running software for your clients (who may be other teams within your company), and your org chart looks like this, your company is probably organized wrong.

The problem is that your org is oriented around job function, and not around products. The problem is Conway’s Law. The org chart shows how you care about information. This org chart shows that you care about job-function status.

Where are the products here? What are your products?

You have people who make things, and people who run things.

Since the people who make things are incentivized to change things, they want frequent change.

Since the people who run things are incentivized to keep things stable, they want infrequent change.

Congratulations, you have two separate groups who do radically different things diametrically opposed to one-another.

<slow clap>

Sure, looking up the org-chart eventually you get high enough that you get to a mutual boss, but what happens then?

They either pick sides, or tell you to "work it out”. Helpful.

The other problem with this is that work is organized in “Projects” or “Initiatives”. Sometimes this is even how “work” is “funded”. Most of the time they’re along the lines of new features along a theme.

Rarely, there may be a “Bug Hunt” or “Security Fixes” initiative, but these are typically when we’re on a waaay-too-old version

< COUGH > Java 1.6 < COUGH >

Or the Support / Customer Support finally complain enough that they get an “appeasement cycle.”

But this pattern of adding things to the pile of things that are being Run by the people who Run things is unsustainable. Sure, you can “just add headcount” to solve some of it, but more new things get created faster than old things are removed.

This is another way of calling out the problem of “Legacy”. I like to define Legacy as “Products that still serve customers, but whose design thinking is no longer current, and probably aren’t actively maintained.”

And no, that pack of “Maintenance Engineers” does not count.

They know you think of them as your C- or D-players, and that their job is to keep the Legacy on life-support.

This is all awful, and entirely preventable. “How?”, I interject on your behalf.

I’ll tell you.

Product Teams.

These are autonomous, largely self-contained teams, who are responsible for solving a set of business problems, do so by caring for a suite of products..

Let’s drill into what I mean by that…

autonomous, largely self-contained teams, who are responsible for solving a set of business problems, do so by caring for a suite of products.

The team is charged with solving a set of problems. Whether it’s handling payments, or an inventory manager, or whatever the business needs: it’s their job to solve.

autonomous, largely self-contained teams, who are responsible for solving a set of business problems, do so by caring for a suite of products.

Whatever solutions are brought to bear to solve the business problem, they are responsible for building and running them.

It bears calling out that the team is responsible for providing a solution to a business problem. It is not necessary that the team build that solution from scratch! If there is a product that already exists that solves the business need, fantastic. They will install and integrate as needed, they will manage relationships with SaaS providers, and monitor that those services are available to serve those business needs.

It would be silly to consider the business problem of “We need to capture metrics and display metrics” and sit down at an IDE in a world where we have not only Graphite and InfluxDB, but also a wealth of SaaS providers to choose from!

autonomous, largely self-contained teams, who are responsible for solving a set of business problems, do so by caring for a suite of products.

The people who are needed to define and solve the problem are on the team. Probably software engineers, maybe some people with QA expertise, probably some release and operations engineering, perhaps Security expertise or DBA.

And this goes beyond Tech. Designer or UX? Perhaps. Maybe Marketing or BA folks. What if you need some Finanace or Payments people?

Get them all on the team!

autonomous, largely self-contained teams, who are responsible for solving a set of business problems, do so by caring for a suite of products.

This is the tricky one.

This is the one that makes middle-management and the architect classes upset. The team is responsible for their own fate. So they get to make their own decisions.

I don’t care what tech you use; whether you’re a Java shop, or only use Node.js for the front-end… or whatever.

The team that is responsible for the app builds the app their way. If they don’t like their way, they change it.

If they want a monolith for one product, and Kubernetes-Dockered Functional Microservices for another, fine.

If they want to rewrite their app in the Hacker-News-Hotness of the week, fine.

But.

It’s a big “But.”

“What about Standards?!” and “What about Interoperability?!” I cry on your behalf…

Internal Company-Wide Standards are overrated. By the time you have a significant number of teams, they’re probably only “followed-ish“.

They might even be causing more harm than good.

What happens if I write something that’s not up to a Reference Architecture Standard?

It ships. Because it didn’t come to light that it wasn’t following standards until it was too late, anyway.

What about Interoperability?!

That’s pretty easily solvable: A Programming Interface. A. P. I.

Document and define the inputs and outputs. Do yourself a favor and version it right from the get-go. As long as you keep to the defined, DOCUMENTED, inputs and outputs, you can change the implementation however you like.

As long as the documentation is correct, and the interface doesn’t change, we’re good! If you need to change the interface, you publish a new version. When nobody is using old versions anymore, you may retire those implementations!

How do Product Teams solve the problem of "Legacy”?

Legacy happens when a project is completed, its resulting product is handed off, and nobody else does significant development anymore.

In the Product Team model, the products don’t get handed off. Since they’re responsible for development and running of the apps, the Product Team keeps the app evergreen.

This can be as simple as updating the app to run on a new version of Java, or not letting the Rails version get too old, etc.

This is where I hear the Directors and Managers of Engineering start getting restless… “What about when I want people to work on something else?”

Well, when the product team has capacity to solve more problems, they can take on solving them with new products. Within the scope of the Product Team.

“Well, what if I just assign people away to another team?”

< low whistle >

Yea, about that… Have a seat, won’t you?

So, under the Product Team structure, the role of Managers and Directors and such changes. Namely, you don’t manage work anymore.

Under a Product Team, the Product Owner is in charge of the work.

The Product Owner is also responsible for hiring to and firing from the product team. They define the problem that the team is to solve, provide context, and set priorities. And assign work to their people.

The role of functional leadership (managers and directors) changes fundamentally. Your job is not the work anymore. I’m going to say that again.

“Your job is not the work anymore.”

Your job is now “Knowledge” and “People.” You coordinate who knows what, and how people are solving problems.

You’re now going to run the internal community of your function in the company. You’re going to get them together every so often to share knowledge and organize conference attendance.

You’re actually going to do 1-on-1’s, and help with people’s career development!

And you’re going to help the Product Owners with their talent. Got engineers who don’t like their product? Offer team swaps.

The example that I like to give is when I need to hire a DBA? I can’t speak for you, but I’d certainly need help with that!

You help the Product Managers with performance reviews, and offer or coordinate mentorship.

What you don’t do is take people from one Product Team and assign them to another.

I’ve worked under that. It’s one thing to have someone quit your team. It’s even worse to have them taken from you.

But as middle-management, is this even what you care about? Maybe you’re more interested in making and running products than taking care of people… Let’s be honest… if you were a kind of engineer who was then “promoted” to management, this could even be likely.

Because, let’s face it, management is a career change, but you probably weren’t given any training or support.

This presents an excellent opportunity to reflect on what management you enjoy, what you’re good at, and what you’d like to pursue!

Closing

In our time together, I’ve described how your org chart is causing pain, but keeping the wrong people numb to it.

I’ve described how organizing around Products ensures that Authority is aligned with Responsibility, while at the same time, solving the problems of “Legacy” and “Unowned” products, preventing useful products from becoming insecure and unmaintainable.

In order to keep the teams stable, we need a new model and agreement of where decisions happen and how management is done.

And finally, we’ve discussed how this model opens up new opportunities for innovation by allowing decisions to be made locally, as well as new prospects for engineers as well as managers to consider the type of work and products that they’re interested in.

Thank you. I hope that you have found this informative, helpful, and entertaining.

Gratitude

I’d like to thank my friends for listening to me rant, and my editor Adam Compton for his help bringing this article together. I’d also like to thank the SysAdvent team for putting in the effort that keeps this fun tradition going.

Contact Me

If you wish to discuss with me further, please feel free to reach out to me. I am gwaldo on Twitter and Gmail/Hangouts and Steam, and seldom refuse hugs (or offers of beverage and company) at conferences.