Day 22 - So, You're Incident Commander, Now What?

By: Joshua Timberman (@jtimberman)

You’re the SRE on call and, while working on a project, your phone buzzes with an alert: “Elevated 500s from API.”

You’re a software developer, and your team lead posts in Slack: “Hey, the library we use for our payment processing endpoint has a remote exploit.”

You work on the customer success team and, during a routine sync with a high-profile customer, they install the new version of your client CLI. Then, when they run any command, it exits with a non-zero return code.

An incident is any situation that disrupts the ability of customers to use a system, service, or software product in a safe and secure manner. And in each of the incidents above, the person who noticed the incident first will most likely become the incident commander. So, now what?

What does it mean to be an incident commander?

Once an individual identifies an incident, one or more people will respond to it. Their goal is to resolve the incident and return systems, services, or other software back to a functional state. While an incident may have a few or many responders—only one person is the incident commander. This role is not about experience, seniority, or position on an org chart; it is to ensure that progress is being made to resolve the incident. The incident commander must think about the inputs from the incident, make decisions about what to do next, and communicate with others about what is happening. The incident commander also determines when the incident is resolved based on the information they have. After the incident is over, the incident commander is also responsible for conducting a post-incident analysis and review to summarize what happened, what the team learned, and what will be done to mitigate the risk of a similar incident happening in the future.

Having a single person—the incident commander—be responsible for handling the incident, delegating responsibility to others, determining when the incident is resolved, and conducting the post-incident review is one of the most effective incident management strategies.

How do you become an incident commander?

Organizations vary on how a team member can become an incident commander. Some call upon the first responder to an incident. Others require specific training and have an on-call rotation of just incident commanders. However you find yourself in the role of incident commander, you should be trusted and empowered by your organization to lead the effort to resolve the incident.

Now what?

Now that you’re incident commander, follow your organizations’ incident response procedure for the specifics about what to do. But for more general questions, we’ve got some guidance.

What are the best strategies for communication and coordination?

One of an incident commander’s primary tasks is to communicate with relevant teams and stakeholders about the status of the incident and to coordinate with other teams to ensure the right people are involved.

If your primary communication tool is Slack, use a separate channel for each incident. Prefix any time-sensitive notes with “timeline” or “TL” so they are easy to find later. If higher-bandwidth communication is required, use a video conference, and keep the channel updated with important information and interactions. When an incident affects external customers, be sure to update them as required by your support teams and agreements with customers.

In the case of a security incident, there may be additional communications requirements with your organization’s legal and/or marketing teams. Legal considerations to communicate may include:

• Statutory or regulatory reporting
• Contractual commitments and obligations to customers
• Insurance claims

Marketing considerations to communicate may include:

• Sensitive information from customer data exposure
• “Zero Day” exploits
• Special messaging requirements, e.g. for publicly traded companies

When should you hand off a long-running incident?

During an extended outage or other long-running incident, you will likely need a break. Whether you are feeling overwhelmed, or that you would contribute better by working on a solution for the incident itself, or that you need to eat, take care of your family, or sleep—all are good reasons to hand off the incident command to someone else.

Coordinate with your other responders in the appropriate channel, whether that’s in a Slack chat or in a Zoom meeting. If necessary, escalate by having someone paged out to get help. Once someone else can take over, communicate with them on the latest progress, the current steps being taken, and who else is involved with the incident. Remember, we’re all human and we need breaks.

How should you approach post-incident analysis and review?

One of an incident commander’s most important jobs is to conduct a post-incident analysis and review after the incident is resolved. This meeting must be blameless: That is, the goal of the meeting is to learn what happened, determine what contributing factors led to the incident, and take action to mitigate the risk of such an incident happening in the future. It’s also to establish a timeline of events, demonstrate an understanding of the problems, and set up the organization for future success in mitigating that problem.

The sooner the incident analysis and review meeting occurs after the incident is resolved, the better. You should ensure adequate rest time for yourself and other responders, but the review meeting should happen within 24 hours—and ideally not longer than two business days after the incident. The incident commander (or commanders) must attend, as they have the most context on what happened and what decisions were made. Any responders who performed significant remediation steps or investigation must also attend so they can share what they learned and what they did during the incident.

Because the systems that fail and cause incidents are complex, a good analysis and review process is complex. Let’s break it down:

Describe the incident

The incident commander will describe the incident. This description should detail the impact as well as its scope, i.e., whether the incident affected internal or external users, how long it took to discover, how long it took to recover, and what major steps were taken to resolve the incident.

“The platform was down” is not a good description.

“On its 5 minute check interval, our monitoring system alerted the on-call engineer that the API service was non-responsive, which meant external customers could not run their workflows for 15 minutes until we were able to restart the message queue” is a good description.

Contributing factors

Successful incident analysis should identify the contributing factors and places where improvements can be made to systems and software. Our world is complex, and technology stacks have multiple moving parts and places where failures occur. Not only can a contributing factor be something technical like “a configuration change was made to an application,” it can be nontechnical like “the organization didn’t budget for new hardware to improve performance.” In reviewing the incident for contributing factors, incident commanders and responders are looking for areas for improvement in order to identify potential corrective actions.

Corrective action items

Finally, incident analysis should determine corrective action items. These must be specific work items that are assigned to a person or a team accountable for their completion, and they must be the primary work priority for that person or team. These aren’t “nice to have,” these are “must do to ensure the safe and reliable operation of the site or service.” Such tasks aren’t necessarily the actions taken during the incident to stabilize or remediate a problem, which are often temporary workarounds to restore service. A corrective action can be as simple as adding new monitoring alerts or system metrics that weren’t implemented before. It can also be as complex as rebuilding a database cluster with a different high availability strategy or migrating to a different database service entirely.

Conclusion

If you’ve recently been the incident commander for your first incident—congratulations. You’ve worked to solve a hard problem that had a lot of moving parts. You took on the role and communicated with the relevant teams and stakeholders. Then, you got some much needed rest and conducted a successful post-incident analysis. Your team identified corrective actions, and your site or service is going to be more reliable for your customers.

Incident management is one of the most stressful aspects of operations work for DevOps and SRE professionals. The first time you become an incident commander, it may be confusing or upsetting. Don’t panic. You’re doing just fine, and you’ll keep getting better.

Further reading

If you’re new to post incident analysis and review, check out Howie: The Post-Incident Guide from Jeli.

PagerDuty also has extensive documentation on incident response and incident command.

Day 15 - Introduction to the PagerDuty API

By: Mandi Walls (@lnxchk)
Edited by: Joe Block (@curiousbiped)

Keeping track of all the data generated by a distributed ecosystem is a daunting task. When something goes wrong, or a service isn’t behaving properly, tracking down the culprit and getting the right folks enabled to fix it is also challenging. PagerDuty can help you with these challenges.

The PagerDuty platform integrates with over 600 other components to gather data, add context, and process automation. Under the hood of all of these integrations is the PagerDuty API, ready to help you programmatically interact with your PagerDuty account.

What’s Exposed Via the API

The PagerDuty API provides access to all the structural objects in your PagerDuty account - users, teams, services, escalation policies, etc - and also to the data objects including incidents, events, and change events.

For objects like users, teams, escalation policies, schedules, and services, you may find using the PagerDuty Terraform Provider will help you maintain the state of your account more efficiently without using the API directly.

The other object types in PagerDuty are more useful when we can send them anytime from anywhere, including via the API from our own code. Let’s take a look at three of them: incidents, events, and change events. If you’d like a copy of the code for these examples, you can find them on Github.

API Basics

To write new information into PagerDuty via the API, you'll need some authorization. You can use OAuth, or create an API key. There are account-level and user-level API keys available. You'll use the account-level keys for the rest of the examples here and keep things simple.

To create a key in your PagerDuty app, you'll need Admin, Global Admin, or Account Owner access to your account. More on that here.

In PagerDuty, navigate to Integrations and then chose API Access Keys. Create a new key, give it a description, and save it somewhere safe. The keys are strings that look like y_NbAkKc66ryYTWUXYEu.

Now you’re ready to generate some incidents! These examples use curl, but there are a number of client libraries for the API as well.

Incidents

Incidents are probably what you’re most familiar with in PagerDuty - they represent a problem or issue that needs to be addressed and resolved. Sometimes this includes alerting a human responder. Many of the integrations in the PagerDuty ecosystem generate incidents from other systems and services to send to PagerDuty.

In PagerDuty, incidents are assigned explicitly to services in your account, so an incoming incident will register with only that service. If your database has too many long-running queries, you want an incident to be assigned to the PagerDuty service representing that database so responders have all the correct context to fix the issue.

If you have a service that doesn’t have an integration out of the box, you can still get information from that service into PagerDuty via the API, and you don’t need anything special to do it. You can send an incident to the API via a curl request to the https://api.pagerduty.com/incidents endpoint.

There are three required headers for these requests, Accept, Content-Type and From, which needs to be an email address associated with your account, for attribution of the incident. Setting up the request will look something like:

curl -X POST --header 'Content-Type: application-json' \
--url https://api.pagerduty.com/incidents \
--header 'Accept: application/vnd.pagerduty+json;version=2' \
--header 'Authorization: Token token=y_NbAkKc66ryYTWUXYEu' \
--header 'From: system2@myemail.com' \

Now you need the information bits of the incident. These will be passed as --data in the curl request. There are just a few required pieces to set up the format and a number of optional pieces that help add context to the incident.

The most important piece you'll need is the service ID. Every object in the PagerDuty platform has a unique identifier. You can find the ID of a service in its URL in the UI. It will be something like https://myaccount.pagerduty.com/service-directory/SERVICEID.

Now you can create the rest of the message with JSON:

curl -X POST --header 'Content-Type: application/json' \
--url https://api.pagerduty.com/incidents \
--header 'Accept: application/vnd.pagerduty+json;version=2' \
--header 'Authorization: Token token=y_NbAkKc66ryYTWUXYEu' \
--header 'From: system2@myemail.com' \
--data '{
  "incident": {
    "type": "incident",
    "title": "Too many blocked requests",
    "service": {
      "id": "PWIXJZS",
      "summary": null,
      "type": "service_reference",
      "self": null,
      "html_url": null
    },
    "body": {
      "type": "incident_body",
      "details": "The service queue is full. Requests are no longer being fulfilled."
    }
  }
}'

When you run this curl command, it will generate a new incident on the service PWIXJZS with the title "To many blocked requests", along with some context in the "body" of the data to help our responders. You can add diagnostics or other information here to help your team fix whatever is wrong.

What if there is information being generated that might not need an immediate response? Instead of an incident, you can create an event.

Events

Events are non-alerting items sent to PagerDuty. They can be processed via Event Rules to help create context on incidents or provide information about the behavior of your services. They utilize the PagerDuty Common Event Format to make processing and collating more effective.

Events are registered to a particular routing_key via an integration on a particular service in your PagerDuty account. In your PagerDuty account, select a service you'd like to send events to, or create new one to practice with. On the page for that service, select the Integrations tab and Add an Integration. For this integration, select "Events API V2" and click Add. You'll have a new integration on your service page. Click the gear icon, and copy the Integration Key. For the full walkthrough of this setup, see the docs.

The next step is to set up the event. The request is a little different from the incident request - the url is different, the From: header is not required, and the authorization is completely handled in the routing_key instead of using an API token.

The content of the request is more structured, based on the Common Event Format, so that you can create event rules and take actions if necessary based on what the events contain.

curl --request POST \
  --url https://events.pagerduty.com/v2/enqueue \
  --header 'Content-Type: application/json' \
  --data '{
  "payload": {
    "summary": "DISK at 99% on machine prod-datapipe03.example.com",
    "timestamp": "2021-11-17T08:42:58.315+0000",
    "severity": "critical",
    "source": "prod-datapipe03.example.com",
    "component": "mysql",
    "group": "prod-datapipe",
    "class": "disk",
    "custom_details": {
      "free space": "1%",
      "ping time": "1500ms",
      "load avg": 0.75
    }
  },
  “event_action”: “trigger”,
  "routing_key": "e93facc04764012d7bfb002500d5d1a6"
}'

Change Events

A third type of contextual data you can send to the API is a Change Event. Change events are non-alerting, and help add context to a service. They are informational data about what's changing in your environment, and while they don't generate an incident, they can inform responders about other activities in the system that might have contributed to a running incident. Change events might come from build and deploy services, infrastructure as code, security updates, or other places that change is generated in your environment.

These events have a similar basic structure to the general events, and the setup with the routing_key is the same, as you can see in the below example. The custom_details can contain anything you want, like the build number, a link to the build report, or the list of objects that were changed during an Infrastructure as Code execution.

Change events have a time horizon. They expire after 90 days in the system, so you aren't looking at old context based on past changes.

curl --request POST \
  --url https://events.pagerduty.com/v2/change/enqueue \
  --header 'Content-Type: application/json' \
  --data '{
  "routing_key": "737ea619db564d41bd9824063e1f6b08",
  "payload": {
    "summary": "Build Success: Increase snapshot create timeout to 30 seconds",
    "timestamp": "2021-11-17T09:42:58.315+0000",
    "source": "prod-build-agent-i-0b148d1040d565540",
    "custom_details": {
      "build_state": "passed",
      "build_number": "220",
      "run_time": "1236s"
    }
  }
}'

Adding Notes

One final fun bit of functionality you can leverage in PagerDuty's API is with notes. Notes are short text entries added to the timeline of an incident. In some integrations, like PagerDuty and Slack, notes will be sent to any Slack channel that is configured to receive updates for an impacted service, making them helpful for responders to coordinate and record activity across different teams.

Notes are associated with a specific incident, so when you are creating a note, the url will include the incident ID. Incident IDs are similar to the other object IDs in PagerDuty in that you can find them from the URL of the incident in the UI. They are longer strings than other objects than the service ID in the examples above.

The content of a note can be anything that might be interesting to the timeline of the incident, like commands that have been run, notifications that have been sent, or additional data and links for responders and stakeholders.

curl --request POST \
  --url https://api.pagerduty.com/incidents/{id}/notes \
  --header 'Accept: application/vnd.pagerduty+json;version=2' \
  --header 'Authorization: Token token=y_NbAkKc66ryYTWUXYEu' \
  --header 'Content-Type: application/json' \
  --header 'From: responder2@myemail.com' \
  --data '{
  "note": {
    "content": "Firefighters are on the scene."
  }
}'

Responders utilizing the UI will see notes in a widget on the incident pag.

Next Steps

Using the API to create tooling where integrations don't yet exist, or for internally-developed services, can help your team stay on top of all the moving parts of your ecosystem when you have an incident. Learn more about the PagerDuty resources available at https://developer.pagerduty.com/. Join the PagerDuty Community to learn from other folks working in PagerDuty, ask questions, and get answers.

Day 1 - Why You Need a Postmortem Process

Written by: Gabe Abinante (@gabinante)
Edited by: Shaun Mouton (@sdmouton)

Why Postmortems?

Failure is inevitable. As engineers building and maintaining complex systems, we likely encounter failure in some form on a daily basis. Not every failure requires a postmortem, but if a failure impacts the bottom line of the business, it becomes important to follow a postmortem process. I say “follow a postmortem process” instead of “do a postmortem”, because a postmortem should have very specific goals designed to prevent future failures in your environment. Simply asking the five whys to try and determine the root cause is not enough.

A postmortem is intended to fill out the sort of knowledge gaps that inevitably exist after an outage:

1. Who was involved / Who should have been involved?
2. Was/is communication good between those parties?
3. How exactly did the incident happen, according to the people who were closest to it?
4. What went well / what did we do right?
5. What could have gone better?
6. What action items can we take from this postmortem to prevent future occurrence?
7. What else did we learn?

Without a systematic examination of failure, observers can resort to baseless speculation.

Without an analysis of what went right as well as what went wrong, the process can be viewed as a complete failure.

Without providing key learnings and developing action items, observers are left to imagine that the problem will almost certainly happen again.

A Case Study of the Knight Capital critical SMARS error 2012

Knight Capital was a financial services firm engaging in high frequency trading in the New York Stock Exchange and NASDAQ. It posted revenue of 1.404 billion in 2011, but went out of business by the end of 2012.

On August 1, 2012, Knight Capital deployed untested software which contained an obsolete function to a production environment. The incident happened due to an engineer deploying new code to only 7/8 of the servers responsible for Knight’s automated routing system for equity orders. The code repurposed a flag that was formerly used to activate an old function known as “Power Peg”, which was designed to move stock prices higher and lower in order to verify the behavior of trading algorithms in a controlled environment. All orders sent with the repurposed flag to one of the servers triggered the obsolete code still present on that server. As a result, Knight’s trading activities caused a major disruption in the prices of 148 companies listed at the New York Stock Exchange. This caused the prices of certain stocks to jump by as much as 1200%. For the incoming parent orders that were processed by the defective code, Knight Capital sent millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes (1). Knight Capital took a pre-tax loss of $440 million. Despite a bailout the day after, this precipitated the collapse of Knight Capital’s stock, losing 75% of their equity value.

I chose to write about this incident because there is an incredible body of writing about it, but actually remarkably little information or substance beyond the SEC release. The amount of material is certainly partially because the incident had such a high impact - few companies have a technical glitch that puts them out of business so quickly. I believe that there’s more to it however - this type of response is an attempt by the community to make sense of the incident because the company itself never released a public postmortem. This is an incredibly interesting case because a production bug and operational failure actually perpetuated the collapse of a seemingly successful business - but the lack of a public postmortem exposed the company to all kinds of baseless speculation about lackadaisical attitudes towards change controls, testing, and production changes (see various citations, especially 11, 12). It would also seem that there was not an internal postmortem, or that it was not well circulated, based upon the Knight Capital CEO’s comments to the press (2).

As the John Allspaw notes in his blog (3), one of the worst consequences of Knight’s reticence was news companies and bloggers using the SEC investigation as a substitute for a postmortem. This was harmful to the business and particularly to the engineers involved in the incident. The SEC document is blamey. It’s supposed to be blamey. It details the incident timeline and outlines procedures that should have been in place to prevent an error - and in doing so it focuses entirely on what was lacking from their outside perspective. What it doesn’t do is accurately explain how the event came to be. What processes WERE in place that the engineers relied upon? What change controls WERE being used? What went right and what will be done to ensure this doesn’t happen in the future?

Did Knight Capital go out of business because they lost a bunch of money in a catastrophic way? Sure. But their core business was still a profitable model - it’s conceivable that they could have received a bailout, continued operations, and gotten out of the hole created by this massive failure. Unfortunately, they failed to demonstrate to their investors and to the public that they were capable of doing so. By failing to release a public document, they allowed the narrative to be controlled by news sites and bloggers.

Taking a look at IaaS provider outages

Infrastructure providers are in a unique position where they have to release postmortems to all of their customers for every outage, because all of their customers’ business systems rely upon IaaS uptime.

AWS experienced an outage that spanned April 21st-April 24th, 2011 and brought down the web infrastructure of several large companies such as Quora and Hootsuite. The incident began when someone improperly executed a network change and shunted a bunch of traffic to the wrong place, which cut a ton of nodes off from each other. Because so many nodes were affected at one time, all of them trying to re-establish replication and hunt for free nodes caused the entire EBS cluster to run out of free space. This generated a cascading failure scenario that required a massive amount of storage capacity in order to untangle. Recovery took quite a while because capacity had to be physically added to the cluster. The postmortem was published via Amazon’s blog on April 29th, 2011. This incident is notable because it was somewhat widespread (affected multiple availability zones) and resolution took longer than 24 hours - making it one of the largest outages that AWS has experienced. AWS has a response pattern that is characterized by communication throughout; updates to status page during the incident, followed by a detailed postmortem afterwards (4). Amazon’s postmortem structure seems to be consistent across multiple events. Many seem to use roughly this outline:

1. Statement of Purpose
2. Infrastructure overview of affected systems
3. Detailed recap of incident by service
4. Explanation of recovery steps & key learnings
5. Wrap-up and conclusion

From this we can learn two things: Firstly, we know that amazon has a postmortem process. They are pursuing specific goals around analyzing the failure of their service. Secondly, we know what they want to communicate. Primarily, they want to explain why the failure occurred and why it will not happen again in the future. They also provide an avenue for disgruntled stakeholders to reach out, receive compensation, get additional explanation, etc.

Azure experienced a similar storage failure in 2014 and we see a similar response from them - immediate communication via status pages, followed by a postmortem after the incident (5).

Taking a look at how the media approaches these failure events, it’s worthy of note that the articles written about the outages include links to the postmortem itself, as well as status pages and social media (6,7). Because the companies are communicative and providing documentation about the problem, the journalist can disseminate that information in their article - thus allowing the company that experienced the failure to control the narrative. Because so much information is supplied, there’s very little speculation about what went right or wrong on the part of individuals or journalists, despite the outage events impacting a huge number of individuals and companies utilizing the services themselves or software which relied upon them.

Conclusion

So, while postmortems are often considered a useful tool only from an engineering perspective, they are critical to all parts of a business for four reasons:

• Resolving existing issues causing failures
• Preventing future failures
• Controlling public perception of the incident
• Helping internal business units and stakeholders to understand the failure

Having a process is equally critical, and that process needs to be informed by the needs of the business both internally and externally. A process helps ensure that the right questions get asked and the right actions are taken to understand and mitigate failure. With the typical software development lifecycle increasing in speed, availability is becoming more of a moving target than ever. Postmortem processes help us zero in on threats to our availability and efficiently attack problems at the source.

About the Author

My name is Gabe Abinante and I am an SRE at ClearSlide. You can reach me at gabe@abinante.com or @gabinante on twitter. If you are interested in developing or refining a postmortem process, check out the Operations Incident Board on GitHub: https://github.com/Operations-Incident-Board

Citations / Additional Reading:

1. http://www.sec.gov/litigation/admin/2013/34-70694.pdf

2. http://spectrum.ieee.org/riskfactor/computing/it/-knuckleheads-in-it-responsible-for-errant-trading-knight-capital-ceo-claims

3. http://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/

4. https://aws.amazon.com/message/65648/

5. https://azure.microsoft.com/en-us/blog/update-on-azure-storage-service-interruption/

6. https://www.wired.com/insights/2011/08/amazon-outage-post-mortem/

7. http://www.rightscale.com/blog/cloud-industry-insights/amazon-ec2-outage-summary-and-lessons-learned

8. http://www.forbes.com/sites/benkepes/2014/11/18/its-a-return-to-the-azure-alypse-microsoft-azure-suffers-widespread-outage/#76bfc0194852

9. http://moneymorning.com/2012/08/07/the-real-story-behind-the-knight-capital-trading-fiasco/

10. https://developer.ibm.com/urbancode/2013/10/28/-knight-capitals-472-million-release-failure/

11. http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172222-a-second-for-45-minutes

12. http://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

13. http://www.cio.com/article/2393212/agile-development/software-testing-lessons-learned-from-knight-capital-fiasco.html

14. http://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/

15. https://codeascraft.com/2016/11/17/debriefing-facilitation-guide/

16. https://github.com/Operations-Incident-Board