Skip to Content
Cloud Engineering Academy
11. Monitoring, Logging & Troubleshooting

Incident response and on-call runbooks: severity, roles, communication and reviews

Design the human side of operations: severity levels, incident roles, communication cadence, runbooks that an engineer can follow at 3 a.m., a semi-automated rollback with Systems Manager Automation, and a blameless post-incident review.

intermediate⏱ 1 h 40 min🟢 FreeSOA-C03DOP-C02SAP-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Define severity levels in terms of customer and business impact, and map each to a response (who is paged, how fast, what is communicated).
  • Name the incident roles (incident commander, operations lead, communications lead, scribe) and explain why one person should not do all of them.
  • Write a runbook with a fixed structure that a tired engineer can follow, and attach it to an alarm.
  • Automate the safest, most repeated runbook step with a Systems Manager Automation runbook that asks for approval.
  • Run a blameless post-incident review that produces owned, dated actions.
  • Explain what changed with AWS Systems Manager Incident Manager and where that leaves this module's design.

Prerequisites

Why this matters in banking

A payments outage at a bank is not only an engineering problem. Customers cannot pay rent, the contact centre phones ring, the business wants updates every fifteen minutes, and the organisation may owe regulators timely information about material operational incidents. Regulators and boards also look at how an organisation handled an incident: how quickly it noticed, how clearly it communicated and how well it learned from it. Which incidents must be reported, to whom and how fast is a question for your compliance function; this lesson gives you the operational scaffolding that makes accurate reporting possible, and does not claim to satisfy any regulation.

Without agreed severity, roles and runbooks, three things reliably go wrong: the wrong people are paged (or everyone is), the engineers fixing the problem are also writing the customer updates, and the same incident repeats because nothing was written down. These are people and process failures, not technology failures, and they are cheap to fix before the next incident.

The idea in plain language

An incident is an unplanned event that degrades a service customers or staff depend on. Incident response is the organised way you detect, coordinate, fix and learn from it.

Four pieces make it work:

  1. Severity says how bad it is and therefore how loudly to respond.
  2. Roles split the work so someone is thinking while others are typing commands.
  3. Runbooks are written, tested instructions for known situations so the on-call person need not improvise.
  4. Reviews turn the incident into improvements.

Analogy: a bank's fire drill. Everyone knows the alarm levels, who the fire marshal is, where to assemble and what the checklist says. Nobody invents the plan in the middle of the fire. Afterwards the marshals review what worked and fix the signage.

A word about terminology, because AWS reuses the word. A runbook in this lesson is a human-readable procedure. AWS Systems Manager Automation runbooks are machine-executable documents (YAML or JSON) that perform steps in your account. The best human runbooks link to, or call, an Automation runbook for the risky or repetitive steps.

What problem the AWS service solves

AWS provides building blocks, not a complete incident process:

NeedAWS building blockNotes
DetectCloudWatch alarms, composite alarms, Application Signals SLO burn-rate alarms, Synthetics canariesCovered earlier in the module
NotifySNS topics, email, SMS, HTTPS endpoints; chat integrationAWS's chat integration was renamed; check the current product name ⚠ VERIFY
Create a work itemAlarm actions can create Systems Manager OpsItems or incidents (see the CloudWatch alarm documentation)Incident creation depends on Incident Manager access, see below
Execute a fixSystems Manager Automation runbooks, Lambda, Step FunctionsAdd approval steps for risky actions
RecordCloudTrail, alarm history (kept 30 days), your incident noteAlarm history is not a long-term record
Page humansA paging product (partner) or SNS to a phoneAWS has no first-party on-call scheduler for new customers, see below

What happened to Incident Manager. AWS Systems Manager Incident Manager stopped accepting new customers on 2025-11-07. Existing customers can keep using it, and AWS has said it will not add new features. AWS's migration guidance points to Systems Manager OpsCenter for operational work items, and to AWS Partner Network solutions for automated paging, escalation and on-call. This module therefore teaches the vendor-neutral design: CloudWatch alarm, SNS, a paging integration of your choice, Automation runbooks and human runbooks. If your organisation already uses Incident Manager, the concepts in this lesson map directly to its response plans, contacts and escalation plans. ⚠ VERIFY the current migration guidance before you choose a tool.

Limits to know:

  • Alarm history lasts 30 days, so copy it into the incident note.
  • An alarm invokes actions only when it changes state, so a long-running alarm does not re-page by itself. Your paging tool needs its own re-notify policy.
  • SNS delivery is not an acknowledgement protocol. "The message was delivered" is not "a human has seen it". Use a paging tool that tracks acknowledgement and escalates.
  • Automation runs with a role you choose. Scope it tightly; it can do whatever the role allows.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Monitor action groupsSNS topics plus alarm actionsAlarm → SNS → subscriber; composite alarms reduce noise
Azure Automation runbooks / Logic AppsSystems Manager Automation runbooks, Lambda, Step FunctionsAutomation documents are declarative YAML or JSON with built-in actions such as aws:executeAwsApi and aws:approve
Azure DevOps / ServiceNow incident recordsSystems Manager OpsCenter OpsItems, or your ITSM toolOpsCenter is lightweight; many banks use ServiceNow
Azure Service Health / status communicationAWS Health Dashboard for provider events; your own status page for customersYou own customer communication
PagerDuty/Opsgenie via action groupsSame tools, fed by SNS or EventBridgeThird-party paging remains the norm
Azure Boards post-mortem work itemsYour tracker, with action owners and datesProcess, not product

Architecture

Security-boundary view: the Automation role is the dangerous part of this diagram. It can change production. It should be assumable only by the Automation service, scoped to specific actions on specific resources, and every execution is recorded in CloudTrail and in the Automation execution history.

Step-by-step implementation

Define severity levels

Use impact, not technology. Four levels are enough. Agree them with the business and write them where the on-call engineer will see them.

SeverityDefinition (examples)ResponseUpdate cadence
Sev1Customers cannot complete core journeys (payments, login, balance) or data is at risk; SLO fast burnPage on-call immediately, incident commander named, bridge open, senior management informedEvery 15 minutes
Sev2Significant degradation or a single important feature down, with workaroundPage on-call, incident channelEvery 30 to 60 minutes
Sev3Minor degradation, no customer-facing impact yetTicket, handled in working hoursOn resolution
Sev4Cosmetic or informationalBacklogNone

When uncertain, declare the higher severity and downgrade later; under-declaring costs more.

Assign roles on the first minute

RoleDoesDoes not
Incident commander (IC)Owns the incident, sets priorities, decides mitigation, keeps the hypothesis listType commands in production
Operations leadInvestigates and executes changes, one at a time, announcing eachWrite customer updates
Communications leadPosts internal and customer-facing updates on the agreed cadence, liaises with the contact centreDebug
ScribeKeeps the timeline: what was seen, decided and done, with timestampsDo anything else

In a small team one person may hold two roles, but never IC plus operations lead on a Sev1: the person with hands on the keyboard cannot keep the big picture.

Write the runbook skeleton

Use a fixed template so every runbook reads the same (see the template in the Complete code section). Each alarm links to its runbook through the alarm description.

Automate the single riskiest repeated step

For most services that is "roll back the last deployment". An Automation runbook with an approval step makes the action consistent and auditable, and leaves the decision with a human.

Establish communication templates and cadence

Pre-write the first message, the update message and the resolution message. Under stress, filling in blanks beats composing prose.

Hold a blameless review within a few working days

Focus on how the system allowed the failure, not who triggered it. Every action has an owner and a date. Track completion.

AWS Console instructions

  • Alarm with a runbook link: Console → CloudWatch → Alarms → select the alarm → Actions → Edit. In the description put the runbook URL on its own line. Add the SNS topic as the In alarm action.
  • SNS topics: Console → SNS → Topics → Create topic (Standard). Create one per severity (hl-sev1-page, hl-sev2-notify). Add subscriptions: email for the lab, an HTTPS endpoint or partner integration for real paging. Confirm the subscription.
  • Automation runbook: Console → Systems Manager → Documents → Create document → Automation. Paste the YAML below. Then Automation → Execute automation to run it, choosing the Owner or Self owned document and providing parameters. ⚠ VERIFY console wording.
  • OpsCenter: Systems Manager → OpsCenter → Create OpsItem for a lightweight work item, if you are not using an external ITSM tool.
  • Test the page: CloudWatch → Alarms → select the alarm → there is no button, so use the CLI set-alarm-state command below.

AWS CLI

# 1) Create the severity topics (one per route) and subscribe an email for the lab.
aws sns create-topic --name hl-sev1-page --tags Key=Project,Value=cea-lab Key=Lab,Value=m11-l13 \
  --profile learn-sandbox --region us-east-1
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:hl-sev1-page \
  --protocol email --notification-endpoint oncall@example.com \
  --profile learn-sandbox --region us-east-1

# 2) Create an alarm whose DESCRIPTION carries the runbook link (this is what responders read first).
aws cloudwatch put-metric-alarm --alarm-name hl-payments-sev1 \
  --alarm-description "SEV1: payments availability burning fast. Runbook: https://wiki.example.com/runbooks/payments-5xx" \
  --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/hl-lab-alb/0123456789abcdef \
  --statistic Sum --period 60 --evaluation-periods 3 --datapoints-to-alarm 3 \
  --threshold 20 --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:hl-sev1-page \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m11-l13 \
  --profile learn-sandbox --region us-east-1

# 3) Test the route end to end by forcing the alarm state (it returns to normal at the next evaluation).
aws cloudwatch set-alarm-state --alarm-name hl-payments-sev1 --state-value ALARM \
  --state-reason "Game day: testing the Sev1 route" --profile learn-sandbox --region us-east-1

# 4) Register and run the Automation runbook (rollback with approval).
aws ssm create-document --name hl-m11-rollback-ecs-service --document-type Automation \
  --document-format YAML --content file://rollback-ecs.yaml \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m11-l13 \
  --profile learn-sandbox --region us-east-1

aws ssm start-automation-execution --document-name hl-m11-rollback-ecs-service \
  --parameters "ClusterName=hl-lab,ServiceName=hl-payments-api,TargetTaskDefinition=hl-payments-api:46,AutomationAssumeRole=arn:aws:iam::123456789012:role/hl-m11-automation-role,ApproverArn=arn:aws:iam::123456789012:role/hl-oncall-ic" \
  --profile learn-sandbox --region us-east-1

# 5) Look at an execution (status, current step, approvals pending).
aws ssm describe-automation-executions --filters Key=DocumentNamePrefix,Values=hl-m11 \
  --profile learn-sandbox --region us-east-1

# 6) Alarm history (kept 30 days) -- copy it into the incident note.
aws cloudwatch describe-alarm-history --alarm-name hl-payments-sev1 --history-item-type StateUpdate \
  --max-records 20 --profile learn-sandbox --region us-east-1

Infrastructure as Code

The stack creates the severity topics, an Automation runbook that rolls an ECS service back after human approval, the IAM role the runbook uses, and an alarm that carries the runbook link.

RunbooksStack.cs
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.SSM;
using Constructs;

namespace Harbourline.Monitoring.Infra;

public sealed class RunbooksStack : Stack
{
    private const string RollbackDocument = """
        schemaVersion: '0.3'
        description: 'Roll an ECS service back to a known-good task definition after human approval.'
        assumeRole: '{{ AutomationAssumeRole }}'
        parameters:
          ClusterName:           { type: String }
          ServiceName:           { type: String }
          TargetTaskDefinition:  { type: String, description: 'family:revision known to be healthy' }
          ApproverArn:           { type: String, description: 'IAM ARN allowed to approve' }
          AutomationAssumeRole:  { type: String }
        mainSteps:
          - name: describeCurrent
            action: aws:executeAwsApi
            inputs:
              Service: ecs
              Api: DescribeServices
              cluster: '{{ ClusterName }}'
              services: ['{{ ServiceName }}']
            outputs:
              - Name: currentTaskDef
                Selector: '$.services[0].taskDefinition'
                Type: String
          - name: approve
            action: aws:approve
            timeoutSeconds: 900
            onFailure: Abort
            inputs:
              NotificationArn: '{{ NotificationTopicArn }}'
              Message: 'Approve rollback of {{ ServiceName }} from {{ describeCurrent.currentTaskDef }} to {{ TargetTaskDefinition }}?'
              MinRequiredApprovals: 1
              Approvers: ['{{ ApproverArn }}']
          - name: rollback
            action: aws:executeAwsApi
            inputs:
              Service: ecs
              Api: UpdateService
              cluster: '{{ ClusterName }}'
              service: '{{ ServiceName }}'
              taskDefinition: '{{ TargetTaskDefinition }}'
          - name: waitSteady
            action: aws:waitForAwsResourceProperty
            timeoutSeconds: 600
            inputs:
              Service: ecs
              Api: DescribeServices
              cluster: '{{ ClusterName }}'
              services: ['{{ ServiceName }}']
              PropertySelector: '$.services[0].deployments[0].rolloutState'
              DesiredValues: ['COMPLETED']
        """;

    public RunbooksStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l13");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");

        var sev1 = new Topic(this, "Sev1", new TopicProps { TopicName = "hl-sev1-page" });
        var sev2 = new Topic(this, "Sev2", new TopicProps { TopicName = "hl-sev2-notify" });

        // The role Automation assumes: only what a rollback needs.
        var automationRole = new Role(this, "AutomationRole", new RoleProps
        {
            RoleName = "hl-m11-automation-role",
            AssumedBy = new ServicePrincipal("ssm.amazonaws.com"),
        });
        automationRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = new[] { "ecs:DescribeServices", "ecs:UpdateService" },
            Resources = new[] { "*" }, // ⚠ learning shortcut: scope to the service ARN in production
        }));
        automationRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
        {
            Actions = new[] { "sns:Publish" },
            Resources = new[] { sev2.TopicArn },
        }));

        new CfnDocument(this, "RollbackRunbook", new CfnDocumentProps
        {
            Name = "hl-m11-rollback-ecs-service",
            DocumentType = "Automation",
            DocumentFormat = "YAML",
            Content = RollbackDocument.Replace("{{ NotificationTopicArn }}", sev2.TopicArn),
            UpdateMethod = "NewVersion",
        });

        // Alarm whose description is the first thing a responder reads.
        var alarm = new Alarm(this, "PaymentsSev1", new AlarmProps
        {
            AlarmName = "hl-payments-sev1",
            AlarmDescription =
                "SEV1: payments 5xx rate high. Runbook: https://wiki.example.com/runbooks/payments-5xx " +
                "| Rollback: SSM Automation hl-m11-rollback-ecs-service",
            Metric = new Metric(new MetricProps
            {
                Namespace = "AWS/ApplicationELB",
                MetricName = "HTTPCode_Target_5XX_Count",
                Statistic = "Sum",
                Period = Duration.Minutes(1),
                DimensionsMap = new Dictionary<string, string> { ["LoadBalancer"] = "app/hl-lab-alb/0123456789abcdef" },
            }),
            Threshold = 20,
            EvaluationPeriods = 3,
            DatapointsToAlarm = 3,
            TreatMissingData = TreatMissingData.NOT_BREACHING,
        });
        alarm.AddAlarmAction(new SnsAction(sev1));
    }
}

The {{ ... }} placeholders inside the raw string are Systems Manager parameter references, not C# interpolation, because the string is a plain raw literal (no $ prefix). The notification topic ARN is substituted at synthesis time.

The Automation steps (aws:executeAwsApi, aws:approve, aws:waitForAwsResourceProperty) and their input names follow the Systems Manager Automation action reference as I remember it. Test the document in a sandbox before depending on it. ⚠ VERIFY

Complete code

The runbook template

Every runbook uses the same sections, in the same order. Store it with the code, in the same repository as the service, so it changes with the service.

runbooks/payments-5xx.md
# Runbook: Payments API 5xx errors (alarm: hl-payments-sev1)

**Owner:** Payments team · **Last tested:** 2026-11-18 · **Review every:** 6 months
**Severity guidance:** Sev1 if more than 1% of POST /payments fail for 5 minutes.

## 1. Symptom and impact
What the alarm means in customer terms: some customers cannot submit payments.

## 2. First five minutes (do in order)
1. Acknowledge the page. Open the incident channel. Name the incident commander.
2. Open the triage dashboard: https://example.com/dashboards/hl-m11-triage
3. Write the symptom sentence (facts only) in the incident note.
4. Check "what changed": last deployment time, last infrastructure change (CloudTrail).
5. Check AWS Health and the partner status page.

## 3. Diagnostic decision table
| Observation | Likely cause | Go to |
|---|---|---|
| ELB 502 + stopped tasks exit 137 | Memory limit / OOM | Section 4A |
| ELB 503 + HealthyHostCount 0 | Failing health check / bad deploy | Section 4B |
| Target 5xx + new fingerprint after deploy | Code defect | Section 4B (roll back) |
| 504 + DB connections at max | Pool exhaustion | performance runbook |

## 4. Fixes
### 4A. Increase memory (change task definition, deploy through the pipeline)
### 4B. Roll back
Run the Automation runbook `hl-m11-rollback-ecs-service`.
**Safety checks before rollback:** the target revision is compatible with the current DB schema; no data migration ran in the last release.

## 5. Verification
Same metrics that fired the alarm back to normal for 15 minutes; no stopped tasks with exit 137.

## 6. Escalation
If not mitigated in 20 minutes: page the engineering manager. If customer data may be affected: invoke the security incident process immediately.

## 7. Communication
Templates below. Update every 15 minutes while Sev1.

## 8. After the incident
Create the review within 3 working days. Update this runbook with anything that was missing.

A runbook linter in C#

Runbooks only help if every alarm links to one. This small tool reads your alarms and reports those whose description lacks a runbook link, or that have no action.

RunbookLint.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.CloudWatch" Version="4.*" />
  </ItemGroup>
</Project>
Program.cs
using System.Text.RegularExpressions;
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;

// Usage: dotnet run -- [name-prefix]   e.g. dotnet run -- hl-
var prefix = args.Length > 0 ? args[0] : "hl-";

using var cw = new AmazonCloudWatchClient(RegionEndpoint.USEast1);

var findings = new List<AlarmFinding>();
string? token = null;

do
{
    var res = await cw.DescribeAlarmsAsync(new DescribeAlarmsRequest
    {
        AlarmNamePrefix = prefix,
        // Without this, composite alarms are not returned.
        AlarmTypes = [AlarmType.MetricAlarm, AlarmType.CompositeAlarm],
        NextToken = token,
    });

    foreach (var a in res.MetricAlarms ?? [])
    {
        findings.AddRange(RunbookRules.Check(a.AlarmName, a.AlarmDescription, a.AlarmActions ?? [], a.ActionsEnabled ?? false));
    }

    foreach (var a in res.CompositeAlarms ?? [])
    {
        findings.AddRange(RunbookRules.Check(a.AlarmName, a.AlarmDescription, a.AlarmActions ?? [], a.ActionsEnabled ?? false));
    }

    token = res.NextToken;
} while (token is not null);

foreach (var f in findings.OrderBy(f => f.Alarm))
{
    Console.WriteLine($"{f.Alarm,-40} {f.Problem}");
}

Console.WriteLine(findings.Count == 0 ? "All alarms have a runbook link and an action." : $"{findings.Count} problem(s).");
return findings.Count == 0 ? 0 : 1; // non-zero exit lets CI fail the build

public sealed record AlarmFinding(string Alarm, string Problem);

public static partial class RunbookRules
{
    public static IEnumerable<AlarmFinding> Check(
        string name, string? description, IReadOnlyCollection<string> actions, bool actionsEnabled)
    {
        if (string.IsNullOrWhiteSpace(description) || !RunbookLink().IsMatch(description))
        {
            yield return new AlarmFinding(name, "description has no 'Runbook: https://...' link");
        }

        if (actions.Count == 0)
        {
            yield return new AlarmFinding(name, "no alarm action: nobody will be told");
        }
        else if (!actionsEnabled)
        {
            yield return new AlarmFinding(name, "actions are disabled");
        }

        if (description is not null && !SeverityPrefix().IsMatch(description))
        {
            yield return new AlarmFinding(name, "description does not start with SEV1/SEV2/SEV3");
        }
    }

    [GeneratedRegex(@"Runbook:\s*https?://\S+", RegexOptions.IgnoreCase)]
    private static partial Regex RunbookLink();

    [GeneratedRegex(@"^\s*SEV[123]\b", RegexOptions.IgnoreCase)]
    private static partial Regex SeverityPrefix();
}

A unit test for the rules needs no AWS account, because RunbookRules.Check is a pure function:

RunbookRulesTests.cs
using Xunit;

public class RunbookRulesTests
{
    [Fact]
    public void Alarm_with_link_action_and_severity_has_no_findings()
    {
        var findings = RunbookRules.Check(
            "hl-ok", "SEV1: x. Runbook: https://wiki.example.com/r", ["arn:aws:sns:us-east-1:111111111111:t"], true);

        Assert.Empty(findings);
    }

    [Fact]
    public void Missing_link_and_action_are_both_reported()
    {
        var findings = RunbookRules.Check("hl-bad", "just a description", [], true).Select(f => f.Problem).ToList();

        Assert.Contains(findings, p => p.Contains("Runbook"));
        Assert.Contains(findings, p => p.Contains("no alarm action"));
    }
}

Communication templates

Incident communication templates
FIRST MESSAGE (within 10 minutes of declaring Sev1/Sev2)
[SEV1] Payments: some customers cannot submit payments.
Started: 10:42 UTC. Impact: approx. 3-6% of payment requests failing. 
Status: investigating. Incident commander: <name>. Next update: 11:00 UTC.

UPDATE
[SEV1] Payments update 2: cause identified as <plain-language cause>. 
Mitigation in progress: <action>. Customer impact: <current>. Next update: <time>.

RESOLUTION
[RESOLVED] Payments: service restored at <time>. Duration: <minutes>. 
Customers affected: <description, not names>. Review scheduled: <date>. 

Code explained line by line

  1. AlarmNamePrefix = prefix limits the scan to your own alarms, so you do not lint account-wide AWS-managed alarms.
  2. AlarmTypes = [MetricAlarm, CompositeAlarm] matters: the DescribeAlarms API returns only metric alarms unless you ask for composite alarms as well.
  3. The do/while loop follows NextToken until all pages are read.
  4. ?? [] handles null collections in SDK v4. ActionsEnabled ?? false handles a nullable boolean.
  5. RunbookRules.Check is a pure function with no AWS types, so it is easily unit-tested without an account. Keep policy logic apart from I/O.
  6. Source-generated regex ([GeneratedRegex]) require the containing class to be partial, which is why RunbookRules is declared static partial.
  7. The tool returns exit code 1 when findings exist, so a CI step (dotnet run -- hl-) can fail a build that adds an alarm with no runbook.
  8. The Automation document's aws:approve step pauses execution until an authorised approver acts, and onFailure: Abort stops the run if approval times out or is denied.

Security considerations

⚠ Learning shortcut — not for production

The lab Automation role uses Resource: "*" for ECS, the approval notification goes to an email subscription, and the test fires an alarm with set-alarm-state. In production, scope the role to the specific service ARN, require the approver to be a named IAM role with MFA, send approval notifications to the paging channel, and rehearse on non-production first.

  • Automation is privileged change. Restrict who can start it with IAM, require approvals for production, and review executions through CloudTrail.
  • Runbooks contain operational detail. Keep them in access-controlled repositories; do not include secrets, only where to find them.
  • Do not leak data in updates. Status messages go to many people. Use counts and descriptions, not names or account numbers.
  • Customer-data incidents change the process. If personal data may be involved, switch to the security incident process (Module 5) and involve the data-protection officer. Notification duties and deadlines under UK GDPR are decisions for Legal and Compliance.
  • Protect the paging path. Test it regularly. An unconfirmed SNS subscription or an expired token silently breaks it.
  • Do not weaken controls during an incident. Break-glass access should be time-limited and reviewed afterwards.

Banking use case

Harbourline's on-call set-up for the payments service:

ItemDecision
RotationOne primary and one secondary, weekly, with a written handover
Page criteriaOnly Sev1 and Sev2 alerts that have a runbook; everything else is a ticket
AcknowledgeWithin 5 minutes or the page escalates to the secondary, then the manager
Roles on Sev1IC and operations lead are different people; a communications lead joins in 10 minutes
CadenceStatus update every 15 minutes on Sev1 until mitigated
Business contactsContact centre lead and product owner added to the incident channel
AfterReview within three working days; actions tracked to completion
Game daysQuarterly rehearsal: pull the alarm, follow the runbook, time the steps

Post-incident review template (blameless):

SectionContent
SummaryThree sentences: what happened, impact, duration
ImpactCustomers affected, journeys affected, SLO budget consumed
TimelineUTC timestamps: first symptom, detection, acknowledgement, mitigation, resolution
DetectionHow we found out; how long after the first symptom
Root cause and contributing factorsThe mechanism, plus the conditions that let it happen
What went wellInclude luck, and say so
What went poorlyProcess and system gaps, never individuals
ActionsEach with owner, due date, tracker link, priority
Lessons for other teamsShort, shareable

Useful measures, handled with care: time to detect, time to acknowledge, time to mitigate and time to resolve. Trend them, but do not use them as targets for individuals, because that encourages hiding incidents or closing them early.

Common mistakes

  • Severity by technology instead of impact. "Database CPU high" is not a severity; "customers cannot pay" is.
  • One person doing everything. The IC also typing commands loses the overview.
  • Runbooks that are never tested. An untested runbook is a guess; test them on game days.
  • Alarms without runbooks. If nobody can say what to do, do not page.
  • Paging on everything. Alert fatigue means the real page is ignored.
  • Blame in reviews. People hide mistakes, and you lose the learning.
  • Actions without owners or dates. They never happen.
  • Treating SNS delivery as acknowledgement. Use a tool that tracks who has the page.
  • Using Incident Manager as the default for new designs. It is closed to new customers.

Troubleshooting

SymptomLikely causeFix
Alarm fired but nobody was pagedSNS subscription unconfirmed, topic policy missing, or alarm action disabledConfirm subscription; check topic policy; describe-alarms for ActionsEnabled
Alarm only pages once during a long outageAlarm actions trigger on state change onlyConfigure re-notify in your paging tool
start-automation-execution fails with access deniedCaller lacks ssm:StartAutomationExecution or iam:PassRole for the roleAdd permissions; check the role's trust policy allows ssm.amazonaws.com
Automation waits forever at approvalApprover ARN not allowed, or notification not deliveredUse a role in Approvers; check the topic; approve with aws ssm send-automation-signal ⚠ VERIFY
Rollback succeeded but errors continueCause was not the deployment (schema, dependency)Return to the hypothesis list; check schema compatibility before rolling back
RunbookLint reports nothing for composite alarmsAlarmTypes not setInclude AlarmType.CompositeAlarm
Review never happensNo owner, no dateMake scheduling the IC's closing task
Different people report different severitiesDefinitions too vagueRewrite with concrete thresholds and examples
Runbook out of dateChanged service, no review cycle"Last tested" date and a 6-month review in the header

Cost and cleanup

ItemLabelEstimate
SNS topics, email notificationsfreeNegligible at lab volume
CloudWatch alarm (1)creditsAbout $0.10 per alarm per month in the examples ⚠ VERIFY
Systems Manager Automation documents and executionsfreeBasic Automation steps are free; charges apply for some features such as long-running or high step counts ⚠ VERIFY
IAM role, runbook files, linterfreeNone

Free alternative: write the severity table, role cards, runbook and review template as documents, run the RunbookLint unit tests locally, and "run" a tabletop exercise with a colleague reading a scenario aloud. No AWS account is needed.

# Remove the lab resources.
aws ssm delete-document --name hl-m11-rollback-ecs-service --profile learn-sandbox --region us-east-1
aws cloudwatch delete-alarms --alarm-names hl-payments-sev1 --profile learn-sandbox --region us-east-1
aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:123456789012:hl-sev1-page --profile learn-sandbox --region us-east-1
aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:123456789012:hl-sev2-notify --profile learn-sandbox --region us-east-1
# (or) cdk destroy RunbooksStack / terraform destroy

# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'

Exercises

  • 🟢 Guided: Take the payments runbook template and write the full runbook for one alarm of your choice, including the diagnostic decision table (at least four rows) and the rollback safety checks. Then run the linter logic against a sample alarm description to confirm it passes.
  • 🟡 Challenge: Add a rule to RunbookRules that checks the alarm description contains an owner (for example Owner: payments-team) and add two unit tests. Then extend the tool to print a count by severity prefix.
  • 🔴 Live problem: Ticket: "Our last Sev1 had four people typing commands and no updates for 40 minutes; the contact centre learned about it from customers." Design the changes: roles, cadence, templates, the person responsible for the contact centre channel, and the first game day. Explain how you would measure whether it worked.

Assignment

Produce an "incident response pack" for Harbourline's payments service: severity table, role cards, one complete runbook, the Automation runbook, the communication templates and the review template. Include a game-day script (a scenario, injected fault, expected detection and the timings you will record) and commit the pack with the linter in CI.

Solutions

Knowledge check

Interview questions

  • How do you decide the severity of an incident? By customer and business impact (journeys affected, data risk, SLO burn), not by the technology involved. If unsure, declare higher and downgrade.
  • What goes into a good runbook? Symptom and impact, a first-five-minutes checklist, a decision table, specific fixes with safety checks, verification, escalation and communication, plus an owner and a last-tested date.
  • How do you reduce alert fatigue? Page only on customer-impacting conditions with a runbook, use composite alarms and SLO burn rates, send the rest to tickets and digests, and review noisy alerts after each rotation.
  • Describe a good post-incident review. Blameless, timeline-based, with root cause and contributing factors, what went well, what went poorly, and owned, dated actions that are tracked.

Certification practice

Completion checklist

  • I can write a severity table based on customer impact
  • I can name the four incident roles and what each does not do
  • I wrote one runbook with the full template and linked it from an alarm
  • I understand the Automation approval flow and the Incident Manager change
  • I can run a blameless review and write owned, dated actions
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Severity is about impact; roles keep thinking and typing apart; runbooks make 3 a.m. response repeatable.
  • Every alarm that pages should link to a runbook, and a linter can enforce it.
  • Automation runbooks with approvals make risky steps consistent and auditable.
  • Alarm actions fire on state changes, so use a paging tool with acknowledgement and escalation.
  • Review without blame, assign owners and dates, and rehearse with game days.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page