Incident response and on-call runbooks: severity, roles, communication and reviews
Design the human side of operations: severity levels, incident roles, communication cadence, runbooks that an engineer can follow at 3 a.m., a semi-automated rollback with Systems Manager Automation, and a blameless post-incident review.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Define severity levels in terms of customer and business impact, and map each to a response (who is paged, how fast, what is communicated).
- Name the incident roles (incident commander, operations lead, communications lead, scribe) and explain why one person should not do all of them.
- Write a runbook with a fixed structure that a tired engineer can follow, and attach it to an alarm.
- Automate the safest, most repeated runbook step with a Systems Manager Automation runbook that asks for approval.
- Run a blameless post-incident review that produces owned, dated actions.
- Explain what changed with AWS Systems Manager Incident Manager and where that leaves this module's design.
Prerequisites
- Troubleshooting production issues methodically is the technical half of this lesson: what you do inside the incident.
- Log aggregation and error monitoring for severity-based alert routing.
- Incident response (Module 5) covers security incidents: containment, evidence and legal notification. This lesson covers operational incidents; the two share roles and tooling and the security lesson takes precedence when an incident might be a breach.
- Alarms and composite alarms and Dashboards and Synthetics canaries for the alerts and the screen a runbook starts from.
- Tools: AWS CLI v2 with the
learn-sandboxSSO profile, .NET 10 SDK.
Why this matters in banking
A payments outage at a bank is not only an engineering problem. Customers cannot pay rent, the contact centre phones ring, the business wants updates every fifteen minutes, and the organisation may owe regulators timely information about material operational incidents. Regulators and boards also look at how an organisation handled an incident: how quickly it noticed, how clearly it communicated and how well it learned from it. Which incidents must be reported, to whom and how fast is a question for your compliance function; this lesson gives you the operational scaffolding that makes accurate reporting possible, and does not claim to satisfy any regulation.
Without agreed severity, roles and runbooks, three things reliably go wrong: the wrong people are paged (or everyone is), the engineers fixing the problem are also writing the customer updates, and the same incident repeats because nothing was written down. These are people and process failures, not technology failures, and they are cheap to fix before the next incident.
The idea in plain language
An incident is an unplanned event that degrades a service customers or staff depend on. Incident response is the organised way you detect, coordinate, fix and learn from it.
Four pieces make it work:
- Severity says how bad it is and therefore how loudly to respond.
- Roles split the work so someone is thinking while others are typing commands.
- Runbooks are written, tested instructions for known situations so the on-call person need not improvise.
- Reviews turn the incident into improvements.
Analogy: a bank's fire drill. Everyone knows the alarm levels, who the fire marshal is, where to assemble and what the checklist says. Nobody invents the plan in the middle of the fire. Afterwards the marshals review what worked and fix the signage.
A word about terminology, because AWS reuses the word. A runbook in this lesson is a human-readable procedure. AWS Systems Manager Automation runbooks are machine-executable documents (YAML or JSON) that perform steps in your account. The best human runbooks link to, or call, an Automation runbook for the risky or repetitive steps.
What problem the AWS service solves
AWS provides building blocks, not a complete incident process:
| Need | AWS building block | Notes |
|---|---|---|
| Detect | CloudWatch alarms, composite alarms, Application Signals SLO burn-rate alarms, Synthetics canaries | Covered earlier in the module |
| Notify | SNS topics, email, SMS, HTTPS endpoints; chat integration | AWS's chat integration was renamed; check the current product name ⚠ VERIFY |
| Create a work item | Alarm actions can create Systems Manager OpsItems or incidents (see the CloudWatch alarm documentation) | Incident creation depends on Incident Manager access, see below |
| Execute a fix | Systems Manager Automation runbooks, Lambda, Step Functions | Add approval steps for risky actions |
| Record | CloudTrail, alarm history (kept 30 days), your incident note | Alarm history is not a long-term record |
| Page humans | A paging product (partner) or SNS to a phone | AWS has no first-party on-call scheduler for new customers, see below |
What happened to Incident Manager. AWS Systems Manager Incident Manager stopped accepting new customers on 2025-11-07. Existing customers can keep using it, and AWS has said it will not add new features. AWS's migration guidance points to Systems Manager OpsCenter for operational work items, and to AWS Partner Network solutions for automated paging, escalation and on-call. This module therefore teaches the vendor-neutral design: CloudWatch alarm, SNS, a paging integration of your choice, Automation runbooks and human runbooks. If your organisation already uses Incident Manager, the concepts in this lesson map directly to its response plans, contacts and escalation plans. ⚠ VERIFY the current migration guidance before you choose a tool.
Limits to know:
- Alarm history lasts 30 days, so copy it into the incident note.
- An alarm invokes actions only when it changes state, so a long-running alarm does not re-page by itself. Your paging tool needs its own re-notify policy.
- SNS delivery is not an acknowledgement protocol. "The message was delivered" is not "a human has seen it". Use a paging tool that tracks acknowledgement and escalates.
- Automation runs with a role you choose. Scope it tightly; it can do whatever the role allows.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Monitor action groups | SNS topics plus alarm actions | Alarm → SNS → subscriber; composite alarms reduce noise |
| Azure Automation runbooks / Logic Apps | Systems Manager Automation runbooks, Lambda, Step Functions | Automation documents are declarative YAML or JSON with built-in actions such as aws:executeAwsApi and aws:approve |
| Azure DevOps / ServiceNow incident records | Systems Manager OpsCenter OpsItems, or your ITSM tool | OpsCenter is lightweight; many banks use ServiceNow |
| Azure Service Health / status communication | AWS Health Dashboard for provider events; your own status page for customers | You own customer communication |
| PagerDuty/Opsgenie via action groups | Same tools, fed by SNS or EventBridge | Third-party paging remains the norm |
| Azure Boards post-mortem work items | Your tracker, with action owners and dates | Process, not product |
Architecture
Security-boundary view: the Automation role is the dangerous part of this diagram. It can change production. It should be assumable only by the Automation service, scoped to specific actions on specific resources, and every execution is recorded in CloudTrail and in the Automation execution history.
Step-by-step implementation
Define severity levels
Use impact, not technology. Four levels are enough. Agree them with the business and write them where the on-call engineer will see them.
| Severity | Definition (examples) | Response | Update cadence |
|---|---|---|---|
| Sev1 | Customers cannot complete core journeys (payments, login, balance) or data is at risk; SLO fast burn | Page on-call immediately, incident commander named, bridge open, senior management informed | Every 15 minutes |
| Sev2 | Significant degradation or a single important feature down, with workaround | Page on-call, incident channel | Every 30 to 60 minutes |
| Sev3 | Minor degradation, no customer-facing impact yet | Ticket, handled in working hours | On resolution |
| Sev4 | Cosmetic or informational | Backlog | None |
When uncertain, declare the higher severity and downgrade later; under-declaring costs more.
Assign roles on the first minute
| Role | Does | Does not |
|---|---|---|
| Incident commander (IC) | Owns the incident, sets priorities, decides mitigation, keeps the hypothesis list | Type commands in production |
| Operations lead | Investigates and executes changes, one at a time, announcing each | Write customer updates |
| Communications lead | Posts internal and customer-facing updates on the agreed cadence, liaises with the contact centre | Debug |
| Scribe | Keeps the timeline: what was seen, decided and done, with timestamps | Do anything else |
In a small team one person may hold two roles, but never IC plus operations lead on a Sev1: the person with hands on the keyboard cannot keep the big picture.
Write the runbook skeleton
Use a fixed template so every runbook reads the same (see the template in the Complete code section). Each alarm links to its runbook through the alarm description.
Automate the single riskiest repeated step
For most services that is "roll back the last deployment". An Automation runbook with an approval step makes the action consistent and auditable, and leaves the decision with a human.
Establish communication templates and cadence
Pre-write the first message, the update message and the resolution message. Under stress, filling in blanks beats composing prose.
Hold a blameless review within a few working days
Focus on how the system allowed the failure, not who triggered it. Every action has an owner and a date. Track completion.
AWS Console instructions
- Alarm with a runbook link: Console → CloudWatch → Alarms → select the alarm → Actions → Edit. In the description put the runbook URL on its own line. Add the SNS topic as the In alarm action.
- SNS topics: Console → SNS → Topics → Create topic (Standard). Create one per severity (
hl-sev1-page,hl-sev2-notify). Add subscriptions: email for the lab, an HTTPS endpoint or partner integration for real paging. Confirm the subscription. - Automation runbook: Console → Systems Manager → Documents → Create document → Automation. Paste the YAML below. Then Automation → Execute automation to run it, choosing the
OwnerorSelf owneddocument and providing parameters. ⚠ VERIFY console wording. - OpsCenter: Systems Manager → OpsCenter → Create OpsItem for a lightweight work item, if you are not using an external ITSM tool.
- Test the page: CloudWatch → Alarms → select the alarm → there is no button, so use the CLI
set-alarm-statecommand below.
AWS CLI
# 1) Create the severity topics (one per route) and subscribe an email for the lab.
aws sns create-topic --name hl-sev1-page --tags Key=Project,Value=cea-lab Key=Lab,Value=m11-l13 \
--profile learn-sandbox --region us-east-1
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:hl-sev1-page \
--protocol email --notification-endpoint oncall@example.com \
--profile learn-sandbox --region us-east-1
# 2) Create an alarm whose DESCRIPTION carries the runbook link (this is what responders read first).
aws cloudwatch put-metric-alarm --alarm-name hl-payments-sev1 \
--alarm-description "SEV1: payments availability burning fast. Runbook: https://wiki.example.com/runbooks/payments-5xx" \
--namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
--dimensions Name=LoadBalancer,Value=app/hl-lab-alb/0123456789abcdef \
--statistic Sum --period 60 --evaluation-periods 3 --datapoints-to-alarm 3 \
--threshold 20 --comparison-operator GreaterThanThreshold --treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:123456789012:hl-sev1-page \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m11-l13 \
--profile learn-sandbox --region us-east-1
# 3) Test the route end to end by forcing the alarm state (it returns to normal at the next evaluation).
aws cloudwatch set-alarm-state --alarm-name hl-payments-sev1 --state-value ALARM \
--state-reason "Game day: testing the Sev1 route" --profile learn-sandbox --region us-east-1
# 4) Register and run the Automation runbook (rollback with approval).
aws ssm create-document --name hl-m11-rollback-ecs-service --document-type Automation \
--document-format YAML --content file://rollback-ecs.yaml \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m11-l13 \
--profile learn-sandbox --region us-east-1
aws ssm start-automation-execution --document-name hl-m11-rollback-ecs-service \
--parameters "ClusterName=hl-lab,ServiceName=hl-payments-api,TargetTaskDefinition=hl-payments-api:46,AutomationAssumeRole=arn:aws:iam::123456789012:role/hl-m11-automation-role,ApproverArn=arn:aws:iam::123456789012:role/hl-oncall-ic" \
--profile learn-sandbox --region us-east-1
# 5) Look at an execution (status, current step, approvals pending).
aws ssm describe-automation-executions --filters Key=DocumentNamePrefix,Values=hl-m11 \
--profile learn-sandbox --region us-east-1
# 6) Alarm history (kept 30 days) -- copy it into the incident note.
aws cloudwatch describe-alarm-history --alarm-name hl-payments-sev1 --history-item-type StateUpdate \
--max-records 20 --profile learn-sandbox --region us-east-1Infrastructure as Code
The stack creates the severity topics, an Automation runbook that rolls an ECS service back after human approval, the IAM role the runbook uses, and an alarm that carries the runbook link.
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.SSM;
using Constructs;
namespace Harbourline.Monitoring.Infra;
public sealed class RunbooksStack : Stack
{
private const string RollbackDocument = """
schemaVersion: '0.3'
description: 'Roll an ECS service back to a known-good task definition after human approval.'
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
ClusterName: { type: String }
ServiceName: { type: String }
TargetTaskDefinition: { type: String, description: 'family:revision known to be healthy' }
ApproverArn: { type: String, description: 'IAM ARN allowed to approve' }
AutomationAssumeRole: { type: String }
mainSteps:
- name: describeCurrent
action: aws:executeAwsApi
inputs:
Service: ecs
Api: DescribeServices
cluster: '{{ ClusterName }}'
services: ['{{ ServiceName }}']
outputs:
- Name: currentTaskDef
Selector: '$.services[0].taskDefinition'
Type: String
- name: approve
action: aws:approve
timeoutSeconds: 900
onFailure: Abort
inputs:
NotificationArn: '{{ NotificationTopicArn }}'
Message: 'Approve rollback of {{ ServiceName }} from {{ describeCurrent.currentTaskDef }} to {{ TargetTaskDefinition }}?'
MinRequiredApprovals: 1
Approvers: ['{{ ApproverArn }}']
- name: rollback
action: aws:executeAwsApi
inputs:
Service: ecs
Api: UpdateService
cluster: '{{ ClusterName }}'
service: '{{ ServiceName }}'
taskDefinition: '{{ TargetTaskDefinition }}'
- name: waitSteady
action: aws:waitForAwsResourceProperty
timeoutSeconds: 600
inputs:
Service: ecs
Api: DescribeServices
cluster: '{{ ClusterName }}'
services: ['{{ ServiceName }}']
PropertySelector: '$.services[0].deployments[0].rolloutState'
DesiredValues: ['COMPLETED']
""";
public RunbooksStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m11-l13");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
var sev1 = new Topic(this, "Sev1", new TopicProps { TopicName = "hl-sev1-page" });
var sev2 = new Topic(this, "Sev2", new TopicProps { TopicName = "hl-sev2-notify" });
// The role Automation assumes: only what a rollback needs.
var automationRole = new Role(this, "AutomationRole", new RoleProps
{
RoleName = "hl-m11-automation-role",
AssumedBy = new ServicePrincipal("ssm.amazonaws.com"),
});
automationRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = new[] { "ecs:DescribeServices", "ecs:UpdateService" },
Resources = new[] { "*" }, // ⚠ learning shortcut: scope to the service ARN in production
}));
automationRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = new[] { "sns:Publish" },
Resources = new[] { sev2.TopicArn },
}));
new CfnDocument(this, "RollbackRunbook", new CfnDocumentProps
{
Name = "hl-m11-rollback-ecs-service",
DocumentType = "Automation",
DocumentFormat = "YAML",
Content = RollbackDocument.Replace("{{ NotificationTopicArn }}", sev2.TopicArn),
UpdateMethod = "NewVersion",
});
// Alarm whose description is the first thing a responder reads.
var alarm = new Alarm(this, "PaymentsSev1", new AlarmProps
{
AlarmName = "hl-payments-sev1",
AlarmDescription =
"SEV1: payments 5xx rate high. Runbook: https://wiki.example.com/runbooks/payments-5xx " +
"| Rollback: SSM Automation hl-m11-rollback-ecs-service",
Metric = new Metric(new MetricProps
{
Namespace = "AWS/ApplicationELB",
MetricName = "HTTPCode_Target_5XX_Count",
Statistic = "Sum",
Period = Duration.Minutes(1),
DimensionsMap = new Dictionary<string, string> { ["LoadBalancer"] = "app/hl-lab-alb/0123456789abcdef" },
}),
Threshold = 20,
EvaluationPeriods = 3,
DatapointsToAlarm = 3,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
alarm.AddAlarmAction(new SnsAction(sev1));
}
}The {{ ... }} placeholders inside the raw string are Systems Manager parameter references, not C# interpolation, because the string is a plain raw literal (no $ prefix). The notification topic ARN is substituted at synthesis time.
The Automation steps (aws:executeAwsApi, aws:approve, aws:waitForAwsResourceProperty) and their input names follow the Systems Manager Automation action reference as I remember it. Test the document in a sandbox before depending on it. ⚠ VERIFY
Complete code
The runbook template
Every runbook uses the same sections, in the same order. Store it with the code, in the same repository as the service, so it changes with the service.
# Runbook: Payments API 5xx errors (alarm: hl-payments-sev1)
**Owner:** Payments team · **Last tested:** 2026-11-18 · **Review every:** 6 months
**Severity guidance:** Sev1 if more than 1% of POST /payments fail for 5 minutes.
## 1. Symptom and impact
What the alarm means in customer terms: some customers cannot submit payments.
## 2. First five minutes (do in order)
1. Acknowledge the page. Open the incident channel. Name the incident commander.
2. Open the triage dashboard: https://example.com/dashboards/hl-m11-triage
3. Write the symptom sentence (facts only) in the incident note.
4. Check "what changed": last deployment time, last infrastructure change (CloudTrail).
5. Check AWS Health and the partner status page.
## 3. Diagnostic decision table
| Observation | Likely cause | Go to |
|---|---|---|
| ELB 502 + stopped tasks exit 137 | Memory limit / OOM | Section 4A |
| ELB 503 + HealthyHostCount 0 | Failing health check / bad deploy | Section 4B |
| Target 5xx + new fingerprint after deploy | Code defect | Section 4B (roll back) |
| 504 + DB connections at max | Pool exhaustion | performance runbook |
## 4. Fixes
### 4A. Increase memory (change task definition, deploy through the pipeline)
### 4B. Roll back
Run the Automation runbook `hl-m11-rollback-ecs-service`.
**Safety checks before rollback:** the target revision is compatible with the current DB schema; no data migration ran in the last release.
## 5. Verification
Same metrics that fired the alarm back to normal for 15 minutes; no stopped tasks with exit 137.
## 6. Escalation
If not mitigated in 20 minutes: page the engineering manager. If customer data may be affected: invoke the security incident process immediately.
## 7. Communication
Templates below. Update every 15 minutes while Sev1.
## 8. After the incident
Create the review within 3 working days. Update this runbook with anything that was missing.A runbook linter in C#
Runbooks only help if every alarm links to one. This small tool reads your alarms and reports those whose description lacks a runbook link, or that have no action.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.CloudWatch" Version="4.*" />
</ItemGroup>
</Project>using System.Text.RegularExpressions;
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
// Usage: dotnet run -- [name-prefix] e.g. dotnet run -- hl-
var prefix = args.Length > 0 ? args[0] : "hl-";
using var cw = new AmazonCloudWatchClient(RegionEndpoint.USEast1);
var findings = new List<AlarmFinding>();
string? token = null;
do
{
var res = await cw.DescribeAlarmsAsync(new DescribeAlarmsRequest
{
AlarmNamePrefix = prefix,
// Without this, composite alarms are not returned.
AlarmTypes = [AlarmType.MetricAlarm, AlarmType.CompositeAlarm],
NextToken = token,
});
foreach (var a in res.MetricAlarms ?? [])
{
findings.AddRange(RunbookRules.Check(a.AlarmName, a.AlarmDescription, a.AlarmActions ?? [], a.ActionsEnabled ?? false));
}
foreach (var a in res.CompositeAlarms ?? [])
{
findings.AddRange(RunbookRules.Check(a.AlarmName, a.AlarmDescription, a.AlarmActions ?? [], a.ActionsEnabled ?? false));
}
token = res.NextToken;
} while (token is not null);
foreach (var f in findings.OrderBy(f => f.Alarm))
{
Console.WriteLine($"{f.Alarm,-40} {f.Problem}");
}
Console.WriteLine(findings.Count == 0 ? "All alarms have a runbook link and an action." : $"{findings.Count} problem(s).");
return findings.Count == 0 ? 0 : 1; // non-zero exit lets CI fail the build
public sealed record AlarmFinding(string Alarm, string Problem);
public static partial class RunbookRules
{
public static IEnumerable<AlarmFinding> Check(
string name, string? description, IReadOnlyCollection<string> actions, bool actionsEnabled)
{
if (string.IsNullOrWhiteSpace(description) || !RunbookLink().IsMatch(description))
{
yield return new AlarmFinding(name, "description has no 'Runbook: https://...' link");
}
if (actions.Count == 0)
{
yield return new AlarmFinding(name, "no alarm action: nobody will be told");
}
else if (!actionsEnabled)
{
yield return new AlarmFinding(name, "actions are disabled");
}
if (description is not null && !SeverityPrefix().IsMatch(description))
{
yield return new AlarmFinding(name, "description does not start with SEV1/SEV2/SEV3");
}
}
[GeneratedRegex(@"Runbook:\s*https?://\S+", RegexOptions.IgnoreCase)]
private static partial Regex RunbookLink();
[GeneratedRegex(@"^\s*SEV[123]\b", RegexOptions.IgnoreCase)]
private static partial Regex SeverityPrefix();
}A unit test for the rules needs no AWS account, because RunbookRules.Check is a pure function:
using Xunit;
public class RunbookRulesTests
{
[Fact]
public void Alarm_with_link_action_and_severity_has_no_findings()
{
var findings = RunbookRules.Check(
"hl-ok", "SEV1: x. Runbook: https://wiki.example.com/r", ["arn:aws:sns:us-east-1:111111111111:t"], true);
Assert.Empty(findings);
}
[Fact]
public void Missing_link_and_action_are_both_reported()
{
var findings = RunbookRules.Check("hl-bad", "just a description", [], true).Select(f => f.Problem).ToList();
Assert.Contains(findings, p => p.Contains("Runbook"));
Assert.Contains(findings, p => p.Contains("no alarm action"));
}
}Communication templates
FIRST MESSAGE (within 10 minutes of declaring Sev1/Sev2)
[SEV1] Payments: some customers cannot submit payments.
Started: 10:42 UTC. Impact: approx. 3-6% of payment requests failing.
Status: investigating. Incident commander: <name>. Next update: 11:00 UTC.
UPDATE
[SEV1] Payments update 2: cause identified as <plain-language cause>.
Mitigation in progress: <action>. Customer impact: <current>. Next update: <time>.
RESOLUTION
[RESOLVED] Payments: service restored at <time>. Duration: <minutes>.
Customers affected: <description, not names>. Review scheduled: <date>. Code explained line by line
AlarmNamePrefix = prefixlimits the scan to your own alarms, so you do not lint account-wide AWS-managed alarms.AlarmTypes = [MetricAlarm, CompositeAlarm]matters: theDescribeAlarmsAPI returns only metric alarms unless you ask for composite alarms as well.- The do/while loop follows
NextTokenuntil all pages are read. ?? []handles null collections in SDK v4.ActionsEnabled ?? falsehandles a nullable boolean.RunbookRules.Checkis a pure function with no AWS types, so it is easily unit-tested without an account. Keep policy logic apart from I/O.- Source-generated regex (
[GeneratedRegex]) require the containing class to bepartial, which is whyRunbookRulesis declaredstatic partial. - The tool returns exit code 1 when findings exist, so a CI step (
dotnet run -- hl-) can fail a build that adds an alarm with no runbook. - The Automation document's
aws:approvestep pauses execution until an authorised approver acts, andonFailure: Abortstops the run if approval times out or is denied.
Security considerations
⚠ Learning shortcut — not for production
The lab Automation role uses Resource: "*" for ECS, the approval notification goes to an email subscription, and the test fires an alarm with set-alarm-state. In production, scope the role to the specific service ARN, require the approver to be a named IAM role with MFA, send approval notifications to the paging channel, and rehearse on non-production first.
- Automation is privileged change. Restrict who can start it with IAM, require approvals for production, and review executions through CloudTrail.
- Runbooks contain operational detail. Keep them in access-controlled repositories; do not include secrets, only where to find them.
- Do not leak data in updates. Status messages go to many people. Use counts and descriptions, not names or account numbers.
- Customer-data incidents change the process. If personal data may be involved, switch to the security incident process (Module 5) and involve the data-protection officer. Notification duties and deadlines under UK GDPR are decisions for Legal and Compliance.
- Protect the paging path. Test it regularly. An unconfirmed SNS subscription or an expired token silently breaks it.
- Do not weaken controls during an incident. Break-glass access should be time-limited and reviewed afterwards.
Banking use case
Harbourline's on-call set-up for the payments service:
| Item | Decision |
|---|---|
| Rotation | One primary and one secondary, weekly, with a written handover |
| Page criteria | Only Sev1 and Sev2 alerts that have a runbook; everything else is a ticket |
| Acknowledge | Within 5 minutes or the page escalates to the secondary, then the manager |
| Roles on Sev1 | IC and operations lead are different people; a communications lead joins in 10 minutes |
| Cadence | Status update every 15 minutes on Sev1 until mitigated |
| Business contacts | Contact centre lead and product owner added to the incident channel |
| After | Review within three working days; actions tracked to completion |
| Game days | Quarterly rehearsal: pull the alarm, follow the runbook, time the steps |
Post-incident review template (blameless):
| Section | Content |
|---|---|
| Summary | Three sentences: what happened, impact, duration |
| Impact | Customers affected, journeys affected, SLO budget consumed |
| Timeline | UTC timestamps: first symptom, detection, acknowledgement, mitigation, resolution |
| Detection | How we found out; how long after the first symptom |
| Root cause and contributing factors | The mechanism, plus the conditions that let it happen |
| What went well | Include luck, and say so |
| What went poorly | Process and system gaps, never individuals |
| Actions | Each with owner, due date, tracker link, priority |
| Lessons for other teams | Short, shareable |
Useful measures, handled with care: time to detect, time to acknowledge, time to mitigate and time to resolve. Trend them, but do not use them as targets for individuals, because that encourages hiding incidents or closing them early.
Common mistakes
- Severity by technology instead of impact. "Database CPU high" is not a severity; "customers cannot pay" is.
- One person doing everything. The IC also typing commands loses the overview.
- Runbooks that are never tested. An untested runbook is a guess; test them on game days.
- Alarms without runbooks. If nobody can say what to do, do not page.
- Paging on everything. Alert fatigue means the real page is ignored.
- Blame in reviews. People hide mistakes, and you lose the learning.
- Actions without owners or dates. They never happen.
- Treating SNS delivery as acknowledgement. Use a tool that tracks who has the page.
- Using Incident Manager as the default for new designs. It is closed to new customers.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Alarm fired but nobody was paged | SNS subscription unconfirmed, topic policy missing, or alarm action disabled | Confirm subscription; check topic policy; describe-alarms for ActionsEnabled |
| Alarm only pages once during a long outage | Alarm actions trigger on state change only | Configure re-notify in your paging tool |
start-automation-execution fails with access denied | Caller lacks ssm:StartAutomationExecution or iam:PassRole for the role | Add permissions; check the role's trust policy allows ssm.amazonaws.com |
| Automation waits forever at approval | Approver ARN not allowed, or notification not delivered | Use a role in Approvers; check the topic; approve with aws ssm send-automation-signal ⚠ VERIFY |
| Rollback succeeded but errors continue | Cause was not the deployment (schema, dependency) | Return to the hypothesis list; check schema compatibility before rolling back |
RunbookLint reports nothing for composite alarms | AlarmTypes not set | Include AlarmType.CompositeAlarm |
| Review never happens | No owner, no date | Make scheduling the IC's closing task |
| Different people report different severities | Definitions too vague | Rewrite with concrete thresholds and examples |
| Runbook out of date | Changed service, no review cycle | "Last tested" date and a 6-month review in the header |
Cost and cleanup
| Item | Label | Estimate |
|---|---|---|
| SNS topics, email notifications | free | Negligible at lab volume |
| CloudWatch alarm (1) | credits | About $0.10 per alarm per month in the examples ⚠ VERIFY |
| Systems Manager Automation documents and executions | free | Basic Automation steps are free; charges apply for some features such as long-running or high step counts ⚠ VERIFY |
| IAM role, runbook files, linter | free | None |
Free alternative: write the severity table, role cards, runbook and review template as documents, run the RunbookLint unit tests locally, and "run" a tabletop exercise with a colleague reading a scenario aloud. No AWS account is needed.
# Remove the lab resources.
aws ssm delete-document --name hl-m11-rollback-ecs-service --profile learn-sandbox --region us-east-1
aws cloudwatch delete-alarms --alarm-names hl-payments-sev1 --profile learn-sandbox --region us-east-1
aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:123456789012:hl-sev1-page --profile learn-sandbox --region us-east-1
aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:123456789012:hl-sev2-notify --profile learn-sandbox --region us-east-1
# (or) cdk destroy RunbooksStack / terraform destroy
# Confirm nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1 --query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided: Take the payments runbook template and write the full runbook for one alarm of your choice, including the diagnostic decision table (at least four rows) and the rollback safety checks. Then run the linter logic against a sample alarm description to confirm it passes.
- 🟡 Challenge: Add a rule to
RunbookRulesthat checks the alarm description contains an owner (for exampleOwner: payments-team) and add two unit tests. Then extend the tool to print a count by severity prefix. - 🔴 Live problem: Ticket: "Our last Sev1 had four people typing commands and no updates for 40 minutes; the contact centre learned about it from customers." Design the changes: roles, cadence, templates, the person responsible for the contact centre channel, and the first game day. Explain how you would measure whether it worked.
Assignment
Produce an "incident response pack" for Harbourline's payments service: severity table, role cards, one complete runbook, the Automation runbook, the communication templates and the review template. Include a game-day script (a scenario, injected fault, expected detection and the timings you will record) and commit the pack with the linter in CI.
Solutions
A good runbook has the eight sections from the template, a decision table that maps observable symptoms to specific sections (for instance ELB 502 with exit 137 to the memory section, 503 with zero healthy hosts to a health-check section), and safety checks before risky actions (schema compatibility before rollback). A passing description reads "SEV1: ... Runbook: https://wiki.example.com/runbooks/..." so all three linter rules (severity prefix, runbook link, action present) pass. If your description starts with "Payments alarm", the severity rule fails; that is the intended nudge to start with the severity.
Add [GeneratedRegex(@"Owner:\s*\S+", RegexOptions.IgnoreCase)] private static partial Regex OwnerTag(); and yield a finding when the description does not match. Tests: one asserting a description containing Owner: payments-team yields no owner finding, and one asserting that a description without it yields a finding whose text contains "Owner". For the severity count, group findings or alarms by the first token of the description with GroupBy and print Key and Count().
Knowledge check
Answer: The commander needs the overall picture: priorities, hypotheses, communication and decisions. Someone focused on the keyboard loses that view. Separate roles also avoid unrecorded, simultaneous changes.
Answer: A human runbook is a written procedure for people; an Automation runbook is a machine-executable YAML or JSON document that performs steps in your account, optionally with approvals. Good human runbooks link to the Automation runbook for risky or repetitive steps.
Answer: Once, when it changes state into ALARM. (Auto Scaling actions are an exception and repeat.) Re-notification must come from your paging tool.
Answer: AWS Systems Manager Incident Manager. AWS points to Systems Manager OpsCenter for operational work items and to partner solutions for paging and on-call. ⚠ VERIFY current guidance.
Answer: They encourage honest reporting of what happened, which produces accurate timelines, real fixes and evidence of learning. Blame makes people hide information, which harms both reliability and regulatory reporting.
Interview questions
- How do you decide the severity of an incident? By customer and business impact (journeys affected, data risk, SLO burn), not by the technology involved. If unsure, declare higher and downgrade.
- What goes into a good runbook? Symptom and impact, a first-five-minutes checklist, a decision table, specific fixes with safety checks, verification, escalation and communication, plus an owner and a last-tested date.
- How do you reduce alert fatigue? Page only on customer-impacting conditions with a runbook, use composite alarms and SLO burn rates, send the rest to tickets and digests, and review noisy alerts after each rotation.
- Describe a good post-incident review. Blameless, timeline-based, with root cause and contributing factors, what went well, what went poorly, and owned, dated actions that are tracked.
Certification practice
An operations team needs a repeatable procedure to roll back an ECS service that includes a required human approval before the change is made, with the execution logged. Which AWS feature BEST fits?
A. A Systems Manager Automation runbook with an aws:approve step.
B. An EC2 Auto Scaling lifecycle hook.
C. An S3 event notification.
D. A Route 53 weighted routing policy.
Answer: A. Automation documents can include approval steps and every execution is logged. B, C and D do not model approval-gated procedures.
A CloudWatch alarm sends an SNS email when a service is unhealthy, but the alarm has been in ALARM for hours and nobody acted. What is the MOST likely reason notifications stopped being noticed?
A. CloudWatch alarms invoke actions only when the alarm state changes, so no repeat notification is sent while it stays in ALARM. B. SNS limits emails to one per day. C. Alarms are disabled after one hour. D. The metric stopped being collected.
Answer: A. Alarm actions fire on state transitions, so ongoing incidents need re-notification from a paging tool. The other statements are not true behaviours.
After an outage, management wants a process that identifies systemic causes and prevents recurrence without discouraging engineers from reporting mistakes. What should the review follow?
A. A blameless post-incident review producing owned, dated action items. B. A disciplinary hearing identifying the engineer responsible. C. No review, since the incident is resolved. D. A review only if the outage exceeded 24 hours.
Answer: A. Blameless reviews encourage openness and produce systemic fixes. B discourages reporting. C and D forgo learning.
Completion checklist
- I can write a severity table based on customer impact
- I can name the four incident roles and what each does not do
- I wrote one runbook with the full template and linked it from an alarm
- I understand the Automation approval flow and the Incident Manager change
- I can run a blameless review and write owned, dated actions
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Severity is about impact; roles keep thinking and typing apart; runbooks make 3 a.m. response repeatable.
- Every alarm that pages should link to a runbook, and a linter can enforce it.
- Automation runbooks with approvals make risky steps consistent and auditable.
- Alarm actions fire on state changes, so use a paging tool with acknowledgement and escalation.
- Review without blame, assign owners and dates, and rehearse with game days.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Troubleshooting production issues methodically: symptom, hypothesis, evidence, fix
Follow a repeatable process from a vague symptom to a proven root cause, using a worked 5xx investigation on an ECS service behind an Application Load Balancer, and build the dashboard and evidence-gathering tool that make the next investigation faster.
Cost monitoring for observability: CloudWatch bills, budgets and anomaly alerts
Understand what makes CloudWatch and related observability services expensive, find your largest log groups and metric sources, cut waste without losing signal, and set up a budget and anomaly detection so a logging mistake is caught in a day rather than at month end.