Day 10 - Assembling Your Year In Review

By: Paige Bernier (@alpacatron3000)
Edited by: Jennifer Davis (@sigje) and Scott Murphy (@ovsage)

Intro

There are a few moments in my career that I have been struck by a story told with data. When I set out as a Site Reliability Engineer into the big wide world I wanted to capture that data storytelling magic and have adapted a presentation I call the “Year in Review”.

My first company had a tradition of taking a moment to pause and review the year by the numbers. The showstopper was the chart showing the amount of data ingested year over year since the founding.

In a single glance that chart conveyed a story that would take hours to tell!

It communicated the incredible efforts the employees took to scale the system to handle ingesting, processing, publishing and storing an ever increasing mountain of data. It illustrated how far the company had come and we were confronted head on with the realization that “what got you here, won’t get you there”.

The biggest impact I have seen comes after the presentation. Discussions from Year in Reviews have sparked sweeping oncall management changes as well as minor, but important, changes in the way developers engage with the SRE team.

Before diving into implementation details, let’s look at why this type of data storytelling is such a powerful tool by examining the core purpose of SRE

The Mission of SRE

The mission of an SRE team is to improve system reliability by facilitating change.

System reliability is the sum of hundreds of decisions humans make when developing, deploying, and maintaining software systems; it is not an intrinsic property¹ of the systems (Patrick O’Connor, 1998). SRE job descriptions tout phrases like “evangelize a DevOps culture” and “influence without authority” acknowledging our roles as change agents.

And as often heard, “change is hard”. As change agents, we are often faced with conflicting priorities, multiple stakeholders internal and external, and fear of the new and unknown.

However, just as often we hear “change is the only constant”. Whether it’s hardware improvements, operating system upgrades, security vulnerability announcements, software dependencies, or the software that we manage as a service, we are constantly monitoring and implementing change.

Combine these two axioms, for extra difficulty:

Ask any engineer who has been forced into a major operating system upgrade when the version of software they’re running requires the previous OS.

As an SRE I often want to make changes across the entire engineering organization such as developing oncall onboarding, ensuring that we are monitoring the customer’s experience, clarifying the lines of responsibility between developers and operators and more!

These types of changes that affect everyone is difficult to effectively implement until two things are true:

• Is there a shared understanding of the current state?
• Is there agreement that the current state needs to change?

This does not mean there needs to be consensus on what changes need to be made!

Is there a shared understanding of the current state?

The answer to this can be a resounding “Yes!” after your Year in Review presentation. Here’s why:

Humans learn best from stories, feelings, senses, and opinions commonly known as qualitative data. Focusing only on these exclusively you risk coming to broad conclusions without nuance or context.

Businesses claim to operate on data, facts and figures, or quantitative data. Focusing purely on the numbers you risk having too many details leading to irrelevant rabbit holes.

In fact, the two seemingly disparate viewpoints aren’t at odds at all. You can even validate findings by using the other category of data.

Feel: “Our monitoring sucks, none of the last 5 pages I got were actionable”

Fact: The primary oncall was paged 5 times out of business hours last week

Finding: Team X is getting paged frequently for non-actionable reasons

Hosting a “Year in Review” means weaving a story using the quantitative data about what occurred in your systems with the qualitative “anec-data” from a human perspective to build a foundation to introduce change.

Is there agreement that the current state needs to change?

This is a more complex endeavor - identifying and implementing change is the hard work of collaborating across teams, roles and competing incentives, motives, and needs. Think of “Year in Review” as a springboard for driving discussion and debate to align on “do we agree something needs to change?”

What does this look like in practice?

At a previous company I heard from engineers and managers alike that the oncall rotations were in need of a shake up. This was an excellent starting place where everyone agreed that there was a problem but was having trouble implementing the necessary changes.

With a goal in mind to identify what exactly the oncall issues were my team tailored a “Year in Review” focused mainly on oncall metrics such as alert noise, hours oncall per engineer, pages received per engineer. Slides illustrated the deluge of alert storms no human could possibly investigate in a given shift and were largely unactionable noise. The impact of not addressing this problem was clear, we were likely missing important signals in the noise and oncalls weren’t able to effectively prioritize their time.

After reviewing the data as a group, my team facilitated a brainstorm to address the barriers to changing the rotations:

• How to handle ownership when multiple teams contribute code?
• What are the “hot potato” services no one feels comfortable owning?
• What services are unofficially owned by a single engineer that needs documentation?
• What is the goal of a low urgency or warning alert?

Based on the main discussion and others in standups and sidebars, my team proposed new team-service ownership and rotations. Several weeks and few rounds of revisions later we merged the PR with our new Terraformed oncall rotations!

DIY “Year in Review”

So, how do you create a “Year in Review” for an SRE team? To start, I typically have a few things in mind about what I think happened and what the data will show. It is fascinating to see where your perception of the system and reality diverge. You can kick off your process by asking a couple of questions:

• What story are you expecting the data to tell?
• What changes do you think need to be made in the next year to improve reliability?

1. Book a meeting with all parties (including engineers, managers, sre, qa, ops, product managers). If there is an existing meeting like an All-Hands or Demo Hour sign up for a presentation slot
2. Kick off a brainstorming session and have participants list out possible changes to include. Such as new features launched or infrastructure expansions to new regions, or even doubling the organization size.
3. Ask teams (including managers)
   1. What data they would find interesting
   2. What data they could contribute from their domain
4. List the company-specific tooling for data sources like:
   1. Version Control
   2. CI/CD
   3. Monitoring
   4. Incident Management
   5. Ticket tracking system
   6. Documentation store
   7. Support ticket system
5. Enlist the help of others to gather the interesting metrics over the past year or year over year. Some suggestions are:
   1. Noisiest alerts
   2. Number of environments
   3. Oncall engineers
   4. Number of services
   5. Ratio of oncall engineer to number of services oncall for
   6. Age of dependencies/libraries
   7. # of hours oncall per person
   8. Number of features launched
   9. # of after hour pages
   10. Ratio of warning alerts to pages
   11. Number of production deploys rolled up by day
   12. Number of open incident AIs
   13. Ingress traffic or other indicator of system load
   14. Most viewed documentation pages
   15. Most search documentation terms
   16. Time to first PR
   17. ….and so much more!
6. Slice and dice the data trying out top 10 lists, total sum, or segment by using whatever constructs your company has such as:
   1. Department
   2. Service
   3. Team
   4. Product Feature
7. Group the data into themed areas “oncall” “production” “onboarding” etc. If you have convinced folks to co-present with you each person can be responsible for presenting a different theme
8. Assemble into a slide deck with one chart per slide to maximize impact
9. Hold the meeting and present your findings,
10. Discuss! In the meeting, after the meeting before the next Year In Review how you interpreted the data compared to others
11. Publish the data and your queries so everyone can explore and answer their own questions

Parting Thoughts

SREs are uniquely suited to facilitate a Year in Review bringing a system-wide perspective on the people, processes, and technology and mission to improve reliability. Keep in mind that much like effecting change, hosting a Year in Review is not a solo effort!

Going solo means you will only capture YOUR thoughts which will almost certainly be tempered by the unique vantage points from others. The more perspectives you invite, the fuller the story of your system will be.

Please share your favorite data storytelling moments or Year in Review stats with me on Twitter at @alpacatron3000

Citation

O’Connor, P. (1998) Standards in reliability and safety engineering [Article]. Elsevier Science Limited, 9 Dec. 2021.

https://www.sciencedirect.com/science/article/abs/pii/S095183209883010X

Notes

---

1. Since the SRE field is still getting established outside of Google, I started to read perspectives from Reliability Engineering in other disciplines. A nugget from Patrick O’Connor’s “Standards in reliability and safety engineering” paper sparked a spicy but important revelation about reliability.

   “Those reliability standards which apply mathematical/ quantitative methods are also based on the inappropriate application of “scientific” thinking. An engineered system or a component has no intrinsic property of reliability, expressible for example as a failure rate. Truly scientifically based properties of systems and components include mass, power output, etc., and these can therefore be predicted and measured with credibility. However, whether a missile or a microcircuit fails depends upon the quality of the design, production, m~nten~ce and use applied to it. These are human contributions, not “scientific”. “ ↩

Day 8 - D&D for SREs

By: Jennifer Davis (@sigje)

In a past life, I was a full-time SRE and a part-time dragonborn paladin named Lorarath. While at work, I supported thousands of systems in collaboration with a team of geeks. Evenings, I tried to survive imaginary disasters and save the world from the sorceress Morgana. I love collaborative games because they plug into some of the real-world emotional responses and social processes critical for successful, meaningful engagement. They provide a place to practice dealing with critical scenarios in a safe place. When you know the stakes are purely imaginary, you're able to look at your efforts from a distance, to gain understanding and enjoy the process of learning and achieving goals together, even when failing. I want to share a couple of insights D&D has given me about my work and how this can help you.

Building your SRE Team … more than just a name.

SRE has many names: Operations, DevOps, Infrastructure engineering, System Admin. It's someone who deploys and runs a highly available, scalable, and secure service that meets business and partner requirements. But what does that mean? Generally, it means someone with a wide-ranging set of skills tackling different challenges at any point in time.

When you first start a campaign in dungeons and dragons, you choose a class to play. This class will then have specializations that you customize based on how you want to play. Next, you build out your character using a character sheet and create a backstory. This character sheet has several abilities and skills. You have several points to allocate to abilities and skills, which grants you additional chances to handle particular events successfully.

In gaming, you collaborate with your team to ensure that you have a well-rounded team often choosing roles to complement the team. You don't want a team of all "magic users" or hack and slashers. Often, we stop at identifying who we are with that single name, whether it's SRE or sysadmin. As an SRE, I depend on a diverse team with varied skills. I am not seeking people with the same expertise or abilities. I'm looking for people with complementary skills who can help accomplish the goals and visions of the team.

Developing your “character sheet”

There is no equivalent to a "character sheet" when it comes to your job. The closest might be equating a resume or LinkedIn profile to a character sheet. Still, these don't align to all of the possible experiences you gain:

• Submitting git pull requests.
• Participating in hackathons.
• Attending training or conferences.
• The myriad of other day-to-day challenges you face.

Additionally, if you don't practice skills in real life, they languish. For example, I haven't touched Solaris in over a decade, and I no longer document it as a skill.

If SRE did have a character sheet, I think three core abilities would be:

Communication, Collaboration, and Confidence. Let's take a closer look at these specializations and the value of spending energy on these areas.

Specialization: Communication

Communication is a fundamental building block to successful character building. As an SRE, I faced various scenarios that required expert communication.

• The first specialty in communication is the number of messages. How often should I remind people about upcoming scheduled maintenance? How often should I reach out to my manager about working on the right thing? How often should my team get together to talk about team tasks?

• The second specialty in communication is the quality of messages. Communication can be visual, written, or oral. Visuals can often convey much more nuanced meaning than repeating the same information in textual format and an underleveraged method.

• The third specialty in communications is effectiveness. Effectiveness is the degree to which your words lead to the desired results. This specialty is the most advanced because effective communication requires an in-depth understanding of the audience and crafting your message as needed.

Specialization: Collaboration

The second core ability is collaboration. In any product or service, you are working on, work needs to be understood, planned, and executed. It doesn't matter who does the work; it just matters that it gets done.

The role I take today doesn't define who I am. If I say, "I'm an SRE at Company," that is just one characteristic of my story and not my identity. Every day as you go into work and tackle your challenge, recognize your special value and what you bring to the team. Rather than adopting and marrying your identity to a specific role, realize some days you take on a role that may be quite different from what you are used to, and that's part of your character development.

There is a distinction between the members of your team and the roles they play. In gaming, you become comfortable speaking on behalf of your character while having a separate, sometimes meta-conversation with your teammates. Social environments seem to tend towards homeostasis, and you (may) naturally ascribe a simplistic narrative to your co-workers' actions. Adopting this awareness that everyone is filling a role on the team that is not representative of everything about the individuals allows you to approach the work to do the impactful work that needs to get done.

In other words, never say, "well, they are just the ROLENAME and can't do that," or "that's not my job."

Specialization: Confidence

The third core ability for your SRE character sheet is confidence. Confidence is about the innate quality that drives you to take risks (or not).

In gaming, sometimes you take the wrong path, or you put your squishy players out front, and they get severely damaged. Mistakes happen. In the "real world," customers do something unexpected. There are bugs in the software, hardware fails, or someone from the team enters the wrong command on the wrong terminal in the production environment.

Collaborative games teach you to fail as a group and rise again while retaining the group cohesion necessary to succeed. Of course, if a teammate really caused you to be captured by a giant spider, you'd probably flip out. Still, across the game board, one has the emotional wiggle-room to behave in a manner that would be laudable in professional situations.

Playing teaches you about exploring challenges with imagination and a sense of play. You have to piece things together while continuing to take action, both keeping in mind the larger game goals and what's immediately on the board at the same time. In addition to this enormous world to explore, there are complex characters (non-playing characters or NPCs) to talk to, and information gathered within each encounter. Be on the lookout for the helpful non-production engineers (NPEs) in your environment, too; while they may not maintain production, they may have valuable information to support you.

Wrapping Up

So, this article inspired you to add some collaborative gaming to your team building, build out your team with complementary skills, or map out the work of the SRE or system administration to a character sheet. Great, beyond the "character sheet," you need the appropriate visualization. By analyzing the particular work items that an individual completed, there could be an incremented "skill" counter. Additional information like git commits, distribution of package management, and incident management APIs could be gathered and glued together to create a way to look at progress over time. That way, you could make sure to spend time on the skills that will improve you in the direction of your choosing.

If you want to try out D&D, check out your local game stores or related groups. Beginner games often provide preconfigured characters that allow you to practice the gameplay without understanding all of the nuances of playing the game.

Day 2 - Reliability as a Product Feature

By: Martin Smith (@martinb3)
Edited by: Jennifer Davis (@sigje)

Abstract

SRE was born out of thinking about reliability as a product feature. However, all of the industry focus in the last few years on things like SLOs and Error Budgets and Production Engineering teams, and others, that constitute "doing SRE," sometimes means teams don’t take advantage of a product-centric approach to reliability these days. And they lose some of the advantages of doing so as a result. This post covers some project maturity levels, some suggestions for thinking about reliability as an SRE engaged in those kinds of projects, as well as what kinds of collaboration might be most successful in driving reliability-as-product-feature in each phase.

A brief history

Site Reliability Engineering, or SRE for short, was born in 2003 out of a need to improve service reliability at Google. Often described as, “an implementation of DevOps,” the practice of SRE aims to treat operations as a software problem that can be addressed through software engineering techniques.

And according to a survey by the DevOps Institute, SRE has truly taken off. This approach has been widely adopted, with 22% of organizations saying they have an SRE team in 2021. This shift can also be seen with the rise of conferences like USENIX’s SREcon which began in 2014, or the release of the popular, “Google SRE book,” a few years later in 2016.

Whether or not your organization has an SRE team that plans work using SLOs and Error Budgets, regularly reduces toil through automation, or has adopted one of the many SRE rules of thumb, the basic premise of what impact SRE can have sometimes gets lost -- that operations is a software problem. Or, shifting the focus back to the customer perspective, that reliability is a product feature that we build.

Having held DevOps Engineer and Site Reliability Engineer roles in the past, and having been a technical lead for SRE teams, I’ve had many opportunities to define the role, activities, and most importantly, the impact of an SRE team. In each case, I’ve found that focusing back on our customers’ experience of reliability has been the most useful framing when speaking to company leaders about SRE team’s, “why,” instead of reciting a long, confusing list of things SREs might do in a quarter. I’ve also found that it’s an easy litmus test for myself to ensure I’m working on the right things at the right time. If I can’t explain how my work affects customer reliability, keeping in mind that reliability for operators usually leads to reliability for customers, it might be a sign that I need to work on something else.

Shifting focus back to product reliability

Shifting the focus from operations and software engineering to talking about reliability as a product feature has some major benefits. First, it helps our organizations better understand what reliability might mean for them and their product(s) -- whether that’s resilience (tolerant of failure), scalability (can function with large volumes of work), observability (understanding internal state from outputs), or security (trust of the system). These are all product capabilities that often aren’t well understood, but fundamentally all matter to customers.

Reliability benefits from product management support (communication with stakeholders, building roadmaps, helping with prioritization and decisions, etc). For example, do you know who your internal stakeholders are for the scalability of your product? What’s on the roadmap for observability over the next 6 months? 2 years? And importantly, what metrics will you collect to be sure you’ve accomplished those goals and delivered on that roadmap? How does it align with other features’ roadmaps? As a friend and former colleague of mine says, “reliability is a product feature whether you devote engineering time to it or not.” If you don’t explicitly plan for that, your customers will implicitly make their own assumptions about your reliability.

Reliability may start to sound like any other product feature, with both internal and external stakeholders, and that’s by design. Making reliability an explicit part of your organizational planning also has many benefits. Thoughtworks’ Technology Radar (Volume 25) from October of this year recommends adoption of this kind of thinking -- that even internal teams should think of themselves as product teams. They also recommend using concepts from the popular Team Topologies book to figure out how to organize these internal teams. In reviewing examples of team structures from the book, many organizations have adopted Simon Wardley’s Pioneer-Settler-Town Planner (or “PST”) framework, too.

Let’s take a look at how one might apply these two ideas (reliability as a product feature, having a specific team profile) to improve the effectiveness of an SRE team.

1. First, there’s no one-size-fits-all approach to improving reliability; different stages of a project will benefit from different kinds of SRE involvement. In this post, I’ll divide products/services into three levels of maturity: beginning, growing, and established.
2. Then, I’ll describe what kinds of SRE work could be most effective at that maturity level, using the PST framework.

Here’s a graphic that explains the PST framework’s three kinds of roles/activities in more detail.

Team Profiles, from blog post Pioneers, Settlers and Town Planners by Simon Wardley

Beginning phase (with Pioneer SREs)

In new projects, there’s often uncertainty and unanswered questions. Small changes in direction could have large future benefits, but experimental work may be completely discarded, too. SREs can drive reliability at this stage by helping teams build prototypes, fail faster, and make agile decisions, all with reliability as a top of mind concern.

Have you ever had a project get close to production/release without thinking about reliability or operational burdens? “Pioneer SREs” can help. They should be part of the team that’s working to deliver a new product development, evaluate vendors, build out proofs of concept, or make major architectural changes. At this stage of a project, any work to “cover” reliability gaps should be identified or entire directions could be changed due to reliability concerns raised by the team.

Embedding in a team building the new product or feature is a great way for SREs to drive reliability early on in these kinds of projects. When teams only consult briefly on reliability or operational concerns, often the final output doesn’t adequately reflect customer or engineering expectations of reliability of the product or operability of the internals.

The success of Pioneer SREs can be measured by looking at how quickly new products or features show up on the roadmap, how quickly vendor implementations happen, or how quickly a project moves from, “exploration,” to, “concrete proposal.”

The largest risk in this phase is having your SRE team end up owners of the system’s reliability, since they helped design it. Hiding the overall reliability of your system from the other developers, behind an SRE team, will typically turn into a situation where the SRE team ends up being treated as an operational team for any product/service problems. Well-scoped embedding engagements can help avoid this problem by emphasizing that embedded SREs are a training resource for the rest of the team to learn, not coverage for the team once the embedding is over.

Growing phase (with Settler SREs)

In this phase, projects are often working to build production-quality infrastructure, launch to customers, or scale to the required audience. SREs can help actually build mature and scalable components from the initial prototypes. They could also level up the engineering organization on how to prepare for any new operational burdens by emphasizing best practices like automating away toil or choosing good SLOs.

Continuing to embed with teams is a great way for SREs to have a hand in the reliability of a nearly-launched product or feature, especially if SREs influence the team to build for observability, scalability, and security into the product. Consulting with teams on production readiness, especially for brand new teams or brand new services, is another way that SREs can ensure that everything reaching production will meet the original reliability requirements of the product, as well as operational best practices (e.g. automation instead of manual database migrations).

At this phase, SRE building and maintaining an idea of Production Readiness is especially important as a product or organization scales. This ensures a consistent approach to reliability across products or services, as well as creates a minimum bar for reliability that must be satisfied. SREs at this stage may even build automation into a pipeline to guarantee minimum scale or ensure resilience on specific failures.

The success of Settler SREs can be measured by looking at how many new services and features are safely being launched into production, as well as examining things like ease of observability (e.g. effective logging, metrics, or monitoring). Success in this phase is also about establishing patterns that make projects successful (e.g. proposal templates). Project retrospectives are a great way to find those patterns as well as improve SRE engagement with the project.

Established phase (with Town Planner SREs)

In this most mature phase, products or services are usually already generally available, and systemic issues like overall architecture or developer tooling are the most likely to impact reliability.

SREs can influence reliability here by identifying and working to resolve systemic reliability issues (e.g. repeated incidents, poor SLO choices, lack of on-call process, etc). Driving continuous improvement is a very common way that SREs influence reliability at this phase.

In addition, SREs can often identify ways to reduce operational burdens or eliminate large scale toil during this phase, whether through technical automation or architecture changes, or through helping teams build process, knowledge, skills, tools and techniques they need for large scale projects to be repeatedly successful and reliable.

This can be a phase where some SREs will feel there’s a stigma associated with doing less technical work, but the impact of this work cannot be overstated -- it’s where SRE can act as a true multiplier as more and more teams and products/services are launched. Examples include running an incident management program, SLA program, On-call Program, Disaster Recovery/Business Continuity planning, or even a Chaos Engineering program. A strategy to address this concern is to pair SREs with a technical program management function (TPM) so that SREs can focus most on the technical aspects of improvement while TPMs can help with the organizational changes needed to improve a process or execute a program.

Measuring the success of Town Planner SREs can be especially tricky. You might look for simple metric improvements like fewer incidents, reduced incident duration, reduced pages, improved SLO targets, or number of DR tests -- but isolating the SRE impact to these kinds of metrics can be difficult. Qualitative feedback from an SRE team’s internal customers is also frequently used to measure success at this stage. The most impactful SREs at this stage tend to cause paradigm shifts for the other development teams, and often even for their own SRE teammates.

Wrapping up

[PST is] how you take a highly effective company and push it [...] towards a continuously adaptive system. May 8th, 2020 @swardley

I hope that the grouping above is useful to readers for structuring work to drive reliability at various levels of product maturity. Reliability-as-a-product-feature isn’t a magic bullet to solve for an organization that doesn’t understand where it fits in the market or what kind of value it delivers, nor will it make a large difference with an unhealthy product management practice that might not know how to develop and drive delivery of a product and its features over time.

As mentioned earlier, there usually isn’t a, “one-size fits all,” approach to driving reliability. You may still need to establish some best practices for your organization such as “Limit toil to 50% of our work” or “Every product feature that goes live must have a reliability review.” Combined with these kinds of rules of thumb, the proposed divisions and strategies above should help focus your team(s) to make the biggest improvement to reliability for your products and services.

In researching this post, it was helpful to review how organizations “do SRE” at various organizations and companies. Continuous improvement was a clear shared trait among them. It’s also worth reviewing the huge amount of content out there about how SRE can effectively collaborate with other teams (e.g. embedding SREs); a poor relationship or failed collaboration with another team can jeopardize all of your efforts.

I invite and encourage you to write about and share your own experiences, both good and bad, focusing on reliability as a first class product feature at your organization. Special thanks to my own SRE team for the many discussions and ideation sessions on how we can best work to drive reliability. And special thanks to Jennifer Davis, Michael Lumsden, David Nolan, Jordan Rinke, and Kerim Satirli for feedback and editing on this post.

Day 8 - Breaking in a New Company as an SRE

By: Amy Tobey (@AlTobey)
Edited By: Tom Purl (@tompurl)

To put it mildly, 2017 has been an interesting year for everyone. My story has its own bumps and advances, including changing jobs twice recently. Changing jobs is a reality many of us face, hopefully voluntarily. It happens infrequently enough that it isn’t something we spend a lot of time talking about. Life’s trials tend to teach us quickly and I hope to share some of what I’ve learned with you.

Backstory

I started the year as an SRE on the CORE team at Netflix. For various reasons, mostly involving stress, I ended up being fired in late June. I only took a couple days to cool off before kicking off interviews. I knew my financial situation would be stable because of the severance package, but I still wanted to have something lined up as soon as possible. Interviewing can sometimes take many weeks or even months so I got started right away.

My first stop was Apple. I knew lots of engineers from my Cassandra days and a friend there had previously pinged me to see if I was interested. I let them know I was available and that process took off.

I also started talking to two other large tech companies, eventually withdrawing from both. One was too far away and had a lengthy study guide for interviews that I was not willing to commit to. The other was going to make an offer but I found their interview process too heavily tilted towards technical ability with little to no discussion of culture or the day-to-day work.

Finally, I interviewed with two smaller infosec companies. On one call we mutually recognized there was no fit while the other moved on to make me a compelling offer.

I eventually went with the Apple offer as it paid more and I knew more people. The difference I didn’t adequately account for was the commute; the security company is remote and Apple involved a long shuttle commute in bay area traffic.

I was only at Apple for 6 weeks and during that time I was trying to experience it with a beginner’s mind. I read docs, code, and listened a lot. In the end, it wasn’t a culture fit for me and the commute was unbearable at 1-2 hours each way, so I quit.

The day after quitting, I set up a call with Tenable and they graciously renewed their offer to me which I accepted immediately. This time I took a couple weeks off because I really wanted to give them my best start.

Day Zero

Since my job is remote, they mailed me my gear. It arrived the Friday before my first day and I took inventory and set it all aside. Some of my accounts had been activated early, but I resisted the urge to start nesting before I was official.

Week 1

I found some bugs with the Windows 10 installation on my laptop on the first day. I spent far too much time dinking with it and finally emailed my managers and IT and got it sorted out within a few hours. I still ended up installing Linux because I found some things about the standard image too constricting.

Besides messing with my laptop, I had a few meetings and spent a lot of time reading docs and code. I spent the remainder getting to know folks over chat and video calls.

Lessons:

• Start using collective speech immediately, i.e. “ours” instead of “yours”.
• Reach out for help with IT problems more quickly, even if the issue at hand appears user-serviceable.
• Reading docs and code in downtime was a great idea.
• Spending time keeping up with chat rooms is a good way to get to know people in a remote job.
• Keep notes about things that are troublesome like my laptop issues and bumps in setting up accounts (more on this later).
• Generate 4KB ssh keys with a good password to impress your colleagues when they copy/past your pubkey.
• Install Linux sooner :)

Week 2

With a working laptop and most of my accounts set up, it was time to start digging in. I got set up on the bastions and started looking at some tickets for our databases. Getting into live machines for analysis helped expose gaps in the new-hire onboarding workflow. Where possible I updated docs and other things were added to my notes.

Lessons:

• Find a ticket to work on right away and give yourself plenty of time. I hit a few snags with access and local conventions that slowed me down.
• Try to figure things out by searching docs and poking.
• Start committing small changes to the team’s primary repos. e.g. formatting, spelling, tests, and minor bug fixes.
• Set a time limit on self-discovery and ask for help.
• I don’t always take notes, but when I do, it’s usually during my first month on the job.

Week 3

For my third week, the company flew me to headquarters for the official orientation. The first day of company overview, culture introduction, and getting to know the other new hires was enjoyable and interesting. The next couple days were largely sales-focused. My manager and one of my peers flew in and sprung me from those days. We went over our ongoing projects and prioritized work while we got a feel for each other. On the last day of the week I started researching projects we prioritized and started coordinating with other SREs on how to approach things.

Lessons:

• Get to know one or two people in your hiring “class”. It’s fun and you’ll probably run into these folks again and again regardless of which department each of you are in.
• Ask about orientation and find out if you really need to attend all of it. At the SaaS companies I’ve experienced there is a lot of content for sales folks (often much higher turnover than other departments) that might not be very relevant to an SRE.
• Remember to install your password manager on your travel device (unless international).

Week 4

In my reviews of the Cassandra setup I found a few things that needed addressing. Rather than handing off the work that needed to be done, I decided to take it on a little before I felt ready. This gave me the opportunity to start committing to our Ansible repos. I wrote a bit of shell code and updated a bunch of playbooks that led me through learning the basics I needed to work with Ansible daily. I got one project done that was blocking progress for dev teams which led to making friends in those teams as I worked with them.

By this time I was starting to get questions from managers and peers about how I was liking the company and people. I was happy to reply in the positive and kept mental notes about specific things people asked.

Lessons:

• Find a project in your wheelhouse to start with. For example, I knew Cassandra or Linux tuning would be a good place for me. Find some familiar ground and use it to launch yourself forward.
• Clearing blocker tickets is an excellent way to make friends in dev teams.
• Remember to give yourself extra time to adapt to the new environment.

Beginner’s Mind

A few years ago when I was stressing out about a talk, Aaron Morton recommended that I read Zen Mind, Beginner’s Mind. It’s an excellent book if you’re interested in Zen of course, but one thing I took away from it is the value of seeing things with a beginner’s mind. Once we’ve acclaimated to a new environment - be it a company, home, city, or vehicle - we are less likely to notice things that are obvious to beginners. I took my notes and data spanning git logs, email, Jira, and even my shell history to create a document with all the things I noticed that were good and those that needed improvement. I chose to keep things as constructive and positive as I could.

Observations:

• Pointed out some technical debt, noting that it was not bad and totally fixable.
• Wrote a quick review of the state of our Ansible repos and procedures.
• Noted some good patterns I noticed in our AWS setup, particularly that we were leveraging IAM and VPC quite well for being relatively new to cloud ops.
• Included a detailed review of the Cassandra clusters since that’s an area of expertise for me.
• Pointed out gaps in operational tooling that we should focus on.
• Gushed about how I saw the company’s stated culture show up in my interactions with people as I onboarded.
• Wrote a bit about my experience with our product and threw out my BHAG for it.
• Recommended improvements for gear & provisioning specific to technical staff.
• Called out the need to simplify our edge architecture (under way!).
• Talked briefly about my orientation experience and recommended some changes for engineering hires.
• Linked some research about IT procedures that needed to updated.
• Celebrated that folks were using my name and pronouns consistently.
• Offered to advise IT folks on some improvements to laptop deploys.

I wrote the doc with my manager, their manager, and my peers in mind. The last I heard it was being passed around the C-levels with positive remarks. Even though I resolved some blockers and deployed a significant upgrade, this document was by far my largest contribution to date. The people I joined were used to their environment and by way of keeping some notes and adding some narrative, I was able to reach a large chunk of the management team and have an impact that benefitted the whole organization.

Wrapup

Starting a new job is stressful regardless of your role. For those of us working in operations and site reliability, the tech work is usually the easy part. Getting acclaimated into the social fabric and pacing yourself for the long haul is something I think we all struggle with. By being methodical and mindful about your approach to your first few weeks, it’s possible to make an impact without the usual stress of being new and vulnerable.

As with any tips of this sort, your experience may vary. Some things that worked for me may also work for you. Above all, find what works for you, make a plan, and communicate, communicate, communicate.

Day 20 - How to set and monitor SLAs

Written by: Emily Chang (emily@datadoghq.com)
Edited by: Ben Cotton (@funnelfiasco)

SLAs give concrete form to a worthy but amorphous goal: you should always be trying to improve the performance and reliability of your services. If you’re maintaining an SLA, collecting and monitoring the right metrics can help you set goals that are meant to improve performance, rather than simply policing it. In this post we’ll walk through the process of collecting data to define reasonable SLAs, and creating dashboards and alerts to help you monitor and maintain them over time.

The ABCs of SLAs, SLOs, and SLIs

Before we go any further, let us first define what the term SLA means within the context of this article. Throughout this post, we will refer to the terms SLA, SLO, and SLI as they are defined in Site Reliability Engineering, a book written by members of Google’s SRE team. In brief:
- SLA: Service Level Agreements are publicly stated or implied contracts with users—either external customers, or another group/team within your organization. The agreement may also outline the economic repercussions (e.g. service credits) that will occur if the service fails to meet the objectives (SLOs) it contains.
- SLO: Service Level Objectives are objectives that aim to deliver certain levels of service, typically measured by one or more Service Level Indicators (SLI).
- SLI: Service Level Indicators are metrics (such as latency or throughput) that indicate how well a service is performing.

In the next section, we will explore the process of collecting and analyzing key SLI metrics that will help us define reasonable SLAs and SLOs.

Collect data to (re)define SLAs and SLOs

Infrastructure monitoring

Maintaining an SLA is difficult, if not impossible, if you don’t have excellent visibility into your systems and applications. Therefore, the first step to maintaining an SLA is to deploy a monitoring platform to make all your systems and applications observable, with no gaps. Every one of your hosts and services should be submitting metrics to your monitoring platform so that when there is a degradation, you can spot the problem immediately, and diagnose the cause quickly.

Whether you are interested in defining external, user-facing SLAs or internal SLOs, you should collect as much data as you can, analyze the data to see what standards you’re currently achieving, and set reasonable goals from there. Even if you’re not able to set your own SLAs, gathering historical performance data may help you make an argument for redefining more reasonable objectives. Generally, there are two types of data you’ll want to collect: customer-facing data (availability/uptime, service response time), and internal metrics (internal application latency).

Collect user-facing metrics to define external SLAs

Synthetic monitoring tools like Pingdom and Catchpoint are widely used to measure the availability and performance of various user-facing services (video load time, transaction response time, etc.). To supplement this data, you’ll probably also want to use an application performance monitoring (APM) tool to analyze the user-facing performance of each of your applications, broken down by its subcomponents.

When it comes to assessing performance metrics, it’s often not enough to simply look at the average values—you need to look at the entire distribution to gain more accurate insights. This is explained in more detail in Site Reliability Engineering: “Using percentiles for indicators allows you to consider the shape of the distribution and its differing attributes: a high-order percentile, such as the 99th or 99.9th, shows you a plausible worst-case value, while using the 50th percentile (also known as the median) emphasizes the typical case.”

For example, let’s say that we wanted to define an SLA that contains the following SLO: In any calendar month period, the user-facing API service will return 99 percent of requests in less than 100 milliseconds. To determine if this is a reasonable SLA, we used Datadog APM to track the distribution of API request latency over the past month, as shown in the screenshot below.

In this example, the distribution indicates that 99 percent of requests were completed in under 161 ms over the past month. This suggests that it may be difficult to fulfill the previously stated SLO without some backend performance enhancements. But you’d probably be able to meet a 250-ms SLO, assuming this month is fairly representative. If you have the data available, it may also be a good idea to query the latency distribution and graph it over longer periods of time, to identify seasonal trends or long-term variations in performance.

Analyze subcomponent metrics to define internal SLOs

Monitoring customer-facing metrics is important for obvious reasons, but collecting and assessing metrics that impact internal services can be just as crucial. Unlike a user-facing SLA, internal SLO violations do not necessarily result in external or economic repercussions. However, SLOs still serve an important purpose—for example, an SLO can help establish expectations between teams within the same organization (such as how long it takes to execute a query that another team’s service depends on).

Below, we captured and graphed an internal application’s average, 95th percentile, and maximum response times over the past month. By collecting and graphing these metrics over a substantial window of time (the past month), we can identify patterns and trends in behavior, and use this information to assess the ongoing viability of our SLOs.

For example, the graph above indicates that the application response time was maxing out around 1 second, while the response time was averaging about 400 ms. Assuming you have leeway to set your own SLO, looking at the full range of values should help guide you toward a more informed and supportable objective.

Internal SLOs can also serve as more stringent versions of external SLAs. In the case of the graph above, if the external SLA contains an SLO that aims to fulfill 95 percent of requests in under 2 seconds within any given month, this team might choose to set its internal SLO to 1.5 seconds. This would hopefully leave the alerted individual(s) enough time to investigate and take action before the external SLA is violated.

Create SLA-focused dashboards

Once you’ve collected metrics from internal and external services and used this data to define SLOs and SLAs, it’s time to create dashboards to visualize their performance over time. As outlined in Datadog’s Monitoring 101 series, preparing dashboards before you need them helps you detect and troubleshoot issues more quickly—ideally, before they degrade into more serious slowdowns or outages.

End user-focused dashboards

The dashboard below provides a general overview of high-level information, including the real-time status and response time of an HTTP check that pings a URL every second. Separate widgets display the average response time over the past 5 minutes, and the maximum response time over the past hour.

Incorporating event correlation into your graphs can also help provide additional context for troubleshooting. As Ben Maurer has explained, Facebook noticed that it recorded fewer internal SLA violations during certain time periods—specifically, when employees were not releasing code.

To spot if your SLA and SLO violations are correlated with code releases or other occurrences, you may want to overlay them as events on your metric graphs. In the screenshot above, the pink bar on the timeseries graph indicates that a code release occurred, which may have something to do with the spike in page load response time that occurred shortly thereafter. We also included a graph that compares today’s average response time to the previous day’s.

Dashboards that dive beneath the surface

While the previous example provided a general overview of user-facing performance, you should also create more comprehensive dashboards to identify and correlate potential issues across subcomponents of your applications.

In the example shown below, we can see a mixture of customer-facing data (API endpoint errors, slowest page load times), and metrics from the relevant underlying components (HAProxy response times, Gunicorn errors). In each graph, code releases have been overlaid as pink bars for correlation purposes.

Alerting on SLAs

For many businesses, not meeting their SLAs is almost as serious a problem as downtime. So in addition to creating informative dashboards, you should set up alerts to trigger at increasing levels of severity as metrics approach internal and external SLO thresholds.

Classify your alerts by urgency

Effective alerting assigns an appropriate notification method based on the level of urgency. Datadog’s Monitoring 101 post on alerting has detailed guidelines for determining the best method of notification (record, notification, or page).

As mentioned earlier, you may want to set an internal SLO that is more aggressive than the objectives in your external SLA, and alert on that value. Any time a metric crosses a “warning” threshold (approaching the internal SLO threshold), it should be recorded as a lighter notification, such as an email or a chat room message. However, if the internal SLO is violated, you should page the person(s) responsible, which will ideally provide the individual(s) with enough time to address the situation before the external SLA is violated.

Watch your resources

In addition to alerting on SLO thresholds, you should also collect comprehensive metrics on underlying resources—everything from low-level (disks, memory) to higher-level components (databases, microservices). If these go unchecked, they can degrade into a state where they will negatively impact end-user experience.

Degradations of resource metrics (for example, database nodes running low on disk space) may not immediately impact customers, but they should be addressed before they ripple into more serious consequences. In this case, the appropriate alert would be a notification (email, chat channel), so that someone can prioritize it in the near future (unless the disk space is very low, in which case it is probably an urgent problem). Whenever a less serious resource issue (for example, an increase in database replication errors) is detected, it should be saved as a record. Even if it eventually resolves on its own, these records can be useful later on, if further investigation is required.

Set informative, actionable alerts

Successful alerts should:
- clearly communicate what triggered the alert
- provide actionable steps to address the situation
- include information about how it was resolved in the past, if applicable

For further reading, check out this other SysAdvent article for sage advice about making alerts more actionable and eliminating unnecessary noise.

Some examples of alerts that you might create to monitor SLAs as well as their underlying resources:
- Your average service response time over the past day exceeds your internal SLO threshold (alert as page)
- An important HTTP check has been failing for the past 2 minutes (alert as page)
- 20 percent of health checks are failing across your HAProxy backends (alert as record if it resolves itself, or email/chat notification if it doesn’t)

Put your SLA strategy in action

If you don’t have a monitoring platform in place, start there. After that you’ll be ready to set up dashboards and alerts that reflect your SLAs and key resources and services that the SLA depends on.