AWS Resilience Hub and game days
Define a resilience policy with RTO and RPO targets, assess an application with AWS Resilience Hub, then run a realistic failover game day and turn the timings into evidence, using a C# policy checker and drill timeline analyser.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.ResilienceHub) · AWS CLI v2 · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Region us-east-1 (DR us-west-2). Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what AWS Resilience Hub does: describe an application, define a resilience policy, run an assessment, act on recommendations, and validate with tests.
- Write a resilience policy with RTO and RPO targets for the four disruption types: application, cloud infrastructure, Availability Zone and Region.
- Read an assessment's estimated workload RTO and RPO, its compliance status and its drift reports.
- Plan and run a game day: scenario, roles, injects, timeline, success criteria and a blameless debrief.
- Convert a drill's timestamps into measured RTO and RPO and compare them with the policy using C#.
Prerequisites
- Chaos engineering with AWS Fault Injection Service for experiments, stop conditions and steady state.
- Database failover across Regions and AWS Backup in depth, the recovery mechanisms a game day exercises.
- The DR strategies and Route 53 failover lessons earlier in this module (see the module overview).
- Module 11: Monitoring, because alarms and runbooks are assessed too.
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK.
Why this matters in banking
Harbourline's disaster-recovery document says "RTO 30 minutes, RPO 5 minutes" for the payments service. An internal audit asks three questions: Who decided those numbers? What evidence says the architecture can meet them? When was it last exercised, and what changed afterwards?
Resilience Hub helps with the second question: it estimates, from the real resource configuration, whether the application can meet the policy, and tells you what is missing (no cross-Region backup, a single-AZ database, no alarm for a key metric). Game days answer the third: a rehearsal where people follow the runbook under time pressure and the clock produces the actual RTO. Together they replace "we believe" with "we measured".
Regulators expect firms to set impact tolerances and to test their ability to remain within them using severe but plausible scenarios. This lesson shows engineering methods that produce evidence for that; it does not make a firm compliant, and your risk and compliance teams define the scenarios and approve the tests.
The idea in plain language
Resilience Hub is a checking service. You tell it what your application is made of (CloudFormation stacks, Terraform state, a resource group, an EKS cluster or a myApplications definition) and what you promise (the resilience policy). It looks at the configuration, estimates how quickly and how completely you could recover from each type of disruption, and lists the gaps with recommended fixes, alarms, tests and standard operating procedures (SOPs).
A game day is a rehearsal: you announce a realistic failure, the on-call team responds as in a real incident, and a scribe records every decision and time. The result is a timeline and a list of improvements.
Analogy: Resilience Hub is the building inspector who reads the plans and says "this stairwell is too narrow for 200 people in 4 minutes". The game day is the fire drill where you time the evacuation. You need both: plans can look fine and fail in practice, and a drill cannot find a design flaw nobody thought to exercise.
What problem each tool solves
Resilience Hub concepts
- Application: a collection of supported resources that Resilience Hub continuously monitors and assesses.
- Application component (AppComponent): a group of related resources that fail together, such as a primary database and its replica. Components have types, for example compute or database.
- Resilience policy: RTO and RPO targets for four disruption types:
- Application (software): infrastructure is healthy but the software misbehaves, for example a bad deployment, a configuration change, data corruption or a failing dependency.
- Cloud infrastructure (hardware): a component fails locally and is fixed by rebooting or replacing it.
- Cloud infrastructure AZ disruption: one or more Availability Zones are unavailable.
- Cloud infrastructure Region incident: one or more Regions are unavailable.
- Policy tiers: a label such as mission critical, critical, important, core services or non-critical that helps you pick sensible targets ⚠ VERIFY the exact tier names.
- Assessment: the analysis comparing the application with the policy. It produces an estimated workload RTO and RPO per disruption type and a compliance status: Policy met, Policy breached, Not assessed or Changes detected.
- Resilience score: a number showing how closely the application follows the recommendations for the policy, alarms, SOPs and tests.
- Drift detection: reports policy drift (a component that met the policy last time and no longer does) and resource drift (resources added or removed).
- Recommendations: architecture changes, alarms, SOPs and FIS experiment templates. Some come as deployable code.
Resilience Hub can also help you create and run FIS experiments to verify resilience against these outage types, and tests for SOPs measure the estimated workload RTO and RPO.
What it cannot do
- It estimates; it does not prove. An estimate from configuration is not a restore. AWS's own wording expects you to maintain and time your recovery SOPs even when the status is Policy met.
- It only sees what you describe. Resources not in the application definition (for example, a third-party API or an on-premises core system) are outside the assessment.
- It covers a list of supported resources. Check the supported resources page ⚠ VERIFY for your stack.
- It costs money. Pricing is per application assessed ⚠ VERIFY the current model; keep sandbox use short.
- It is not a compliance certificate. A Policy met status is a useful engineering signal, not a regulatory outcome.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Well-Architected Review and Azure Advisor reliability recommendations | AWS Resilience Hub assessments | Resilience Hub compares the resources with explicit RTO and RPO targets you define |
| Azure Site Recovery recovery plans and test failover | Recovery SOPs and game days; AWS Elastic Disaster Recovery drills | The "recovery plan" is runbooks plus automation you assemble |
| Azure Chaos Studio experiments | AWS FIS experiments, recommended by Resilience Hub | Experiments can be tied to the policy's disruption types |
| Service Health and Resource Health | AWS Health Dashboard | Health events tell you when to start a response; Resilience Hub does not monitor AWS-wide events |
| Business continuity exercises run by risk teams | Game days run by engineering and operations | In a bank both exist; game days give the technical evidence for the risk exercise |
Architecture
The loop matters more than any single step: assess, fix, rehearse, measure, adjust the policy or the design, assess again. Measured results from the game day should feed back into the policy (maybe the 30-minute RTO is unrealistic) or the design (add warm standby).
Security-boundary view: the assessment role reads resource configuration across the application's stacks, which is sensitive architecture information, so limit it by IAM. Game-day actions use break-glass roles with MFA, run in the DR and production accounts under change approval, and all actions are logged in CloudTrail for the evidence pack.
Step-by-step implementation
Choose the service and set targets with the business
For the payments service: application disruption RTO 10 minutes (rollback), infrastructure 5 minutes (replace a task), AZ 15 minutes, Region 30 minutes with RPO 5 minutes. The targets come from the impact tolerance, not from what is easy.
Create the resilience policy
Define the four disruption types with RTO and RPO in seconds. Keep the policy in IaC or a version-controlled JSON file.
Describe the application
Import resources from the CloudFormation stacks (or a Terraform state file in S3) of the service. Check that every AppComponent has the right resources, including the database and its replicas, and that cross-Region resources appear.
Publish and run an assessment
Publish a version of the application and start an assessment. Wait for it to finish, then read the compliance status and the estimated RTO and RPO.
Triage the recommendations
Group them into must-fix (breaches the policy), should-fix (alarms and SOPs missing) and could-fix. Turn them into tickets with owners.
Plan the game day
Choose a severe but plausible scenario, for example "the primary Region's database is unavailable at 11:00 on a weekday". Write the objectives, success criteria (RTO 30 minutes, RPO 5 minutes), roles, injects, rollback plan and communications. Get approval.
Run it, time it, debrief
Run the scenario, record timestamps for detect, declare, decide, promote, repoint, verify and recover. Hold a blameless debrief within a week, write the improvements and rerun the failed parts.
AWS Console instructions
- Create a policy: Console → AWS Resilience Hub → Resiliency policies → Create resiliency policy. Pick a tier, or define custom targets. Enter RTO and RPO for Application, Cloud infrastructure, Availability Zone and Region disruptions.
- Add an application: Applications → Add application → Describe: choose the input source (CloudFormation stack, Terraform state file, resource group, EKS or myApplications), select the resources, then review AppComponents → Define: attach the policy → Publish.
- Assess: open the application → Assessments → Run resiliency assessment. Review Compliance, the Resilience score, and tabs for Recommendations, Alarms, SOPs and Tests ⚠ VERIFY tab names.
- Drift: the application dashboard shows drift notifications after each assessment.
- Delete: open the application → Actions → Delete. Then delete any policy or experiment you created.
AWS CLI
# 1. Create a policy. Keys are AZ, Hardware, Software and Region; values are in seconds.
aws resiliencehub create-resiliency-policy \
--policy-name hl-m12-payments-policy \
--tier Critical \
--policy '{
"Software": {"rtoInSecs": 600, "rpoInSecs": 300},
"Hardware": {"rtoInSecs": 300, "rpoInSecs": 60},
"AZ": {"rtoInSecs": 900, "rpoInSecs": 60},
"Region": {"rtoInSecs": 1800, "rpoInSecs": 300}
}' \
--tags Project=cea-lab,Lab=m12-resilience-hub \
--region us-east-1 --profile learn-sandbox
# 2. Create the application and attach the policy.
aws resiliencehub create-app --name hl-m12-payments \
--policy-arn arn:aws:resiliencehub:us-east-1:111122223333:resiliency-policy/POLICY_ID \
--region us-east-1 --profile learn-sandbox
# 3. Import resources from a CloudFormation stack into the draft version.
aws resiliencehub import-resources-to-draft-app-version \
--app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
--source-arns arn:aws:cloudformation:us-east-1:111122223333:stack/hl-m12-payments/STACK_ID \
--region us-east-1 --profile learn-sandbox
# 4. Publish a version, then start an assessment.
aws resiliencehub publish-app-version \
--app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
--region us-east-1 --profile learn-sandbox
aws resiliencehub start-app-assessment \
--app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
--app-version release --assessment-name hl-m12-first \
--region us-east-1 --profile learn-sandbox
# 5. Read the result: compliance status and estimated RTO/RPO per disruption type.
aws resiliencehub describe-app-assessment \
--assessment-arn arn:aws:resiliencehub:us-east-1:111122223333:app-assessment/ASSESSMENT_ID \
--query "assessment.{Status:assessmentStatus,Compliance:complianceStatus,Score:resiliencyScore.score}" \
--region us-east-1 --profile learn-sandbox
# 6. List recommendations and alarm recommendations.
aws resiliencehub list-app-component-recommendations \
--assessment-arn arn:aws:resiliencehub:us-east-1:111122223333:app-assessment/ASSESSMENT_ID \
--region us-east-1 --profile learn-sandbox
# 7. Cleanup: delete the application and policy.
aws resiliencehub delete-app --app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
--force-delete --region us-east-1 --profile learn-sandbox
aws resiliencehub delete-resiliency-policy \
--policy-arn arn:aws:resiliencehub:us-east-1:111122223333:resiliency-policy/POLICY_ID \
--region us-east-1 --profile learn-sandboxIDs and the account number are placeholders. Command and parameter names (--source-arns, --force-delete, the tier names and the policy keys) follow the Resilience Hub API at the time of writing, but they have changed between releases; run aws resiliencehub help and check each subcommand before relying on them ⚠ VERIFY.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.ResilienceHub;
using Constructs;
namespace Harbourline.Ha.Infra;
public sealed class ResiliencePolicyStack : Stack
{
public ResiliencePolicyStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
// The policy as code: targets are reviewed in a pull request like any other change.
var policy = new CfnResiliencyPolicy(this, "PaymentsPolicy", new CfnResiliencyPolicyProps
{
PolicyName = "hl-m12-payments-policy",
Tier = "Critical",
Policy = new Dictionary<string, object>
{
["Software"] = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 600, RpoInSecs = 300 },
["Hardware"] = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 300, RpoInSecs = 60 },
["AZ"] = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 900, RpoInSecs = 60 },
["Region"] = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 1800, RpoInSecs = 300 }
},
Tags = new Dictionary<string, string>
{
["Project"] = "cea-lab", ["Lab"] = "m12-resilience-hub", ["Environment"] = "learn"
}
});
_ = new CfnOutput(this, "PolicyArn", new CfnOutputProps { Value = policy.AttrPolicyArn });
}
}Resilience Hub applications can also be declared with CfnApp, but the application template body and resource mappings are detailed JSON; for the lab, importing from a deployed stack with the CLI or console is simpler ⚠ VERIFY CfnResiliencyPolicy property shapes in your CDK version.
Complete code
Two pure C# tools: a policy checker that compares policy targets with measured or estimated values and reports the gap, and a game-day timeline analyser that turns timestamps into measured RTO and RPO.
namespace Harbourline.Ha.GameDay;
public enum Disruption { Software, Hardware, AvailabilityZone, Region }
public sealed record Target(TimeSpan Rto, TimeSpan Rpo);
public sealed record Observed(TimeSpan Rto, TimeSpan Rpo, string Source); // "assessment estimate" or "game day"
public sealed record PolicyFinding(Disruption Disruption, bool Met, TimeSpan RtoGap, TimeSpan RpoGap, string Source);
public static class PolicyChecker
{
public static IReadOnlyList<PolicyFinding> Check(
IReadOnlyDictionary<Disruption, Target> policy,
IReadOnlyDictionary<Disruption, Observed> observed)
{
var findings = new List<PolicyFinding>();
foreach (var (disruption, target) in policy)
{
if (!observed.TryGetValue(disruption, out var o))
{
// No evidence at all is a failure: untested means not met.
findings.Add(new PolicyFinding(disruption, false, TimeSpan.MaxValue, TimeSpan.MaxValue, "no evidence"));
continue;
}
var rtoGap = o.Rto - target.Rto; // positive means slower than the target
var rpoGap = o.Rpo - target.Rpo; // positive means more data loss than the target
findings.Add(new PolicyFinding(disruption,
Met: rtoGap <= TimeSpan.Zero && rpoGap <= TimeSpan.Zero,
RtoGap: rtoGap, RpoGap: rpoGap, Source: o.Source));
}
return findings;
}
}namespace Harbourline.Ha.GameDay;
public sealed record DrillEvent(DateTimeOffset At, string Step, string Who, string Note = "");
public sealed record DrillSummary(
TimeSpan DetectToDeclare,
TimeSpan DeclareToRecovered,
TimeSpan MeasuredRto, // from incident start to service verified
TimeSpan MeasuredRpo, // data-loss window reported by the database team
IReadOnlyList<(string Step, TimeSpan Duration)> StepDurations);
public static class DrillTimeline
{
// Steps in required order; the analyser refuses a timeline that skips or reorders them.
public static readonly string[] RequiredSteps =
["FaultInjected", "Detected", "Declared", "Promoted", "Repointed", "Verified"];
public static DrillSummary Analyse(IEnumerable<DrillEvent> events, TimeSpan measuredRpo)
{
var ordered = events.OrderBy(e => e.At).ToList();
var byStep = ordered.GroupBy(e => e.Step).ToDictionary(g => g.Key, g => g.First().At);
foreach (var step in RequiredSteps)
if (!byStep.ContainsKey(step))
throw new InvalidOperationException($"Timeline is missing the required step '{step}'.");
for (var i = 1; i < RequiredSteps.Length; i++)
if (byStep[RequiredSteps[i]] < byStep[RequiredSteps[i - 1]])
throw new InvalidOperationException(
$"Step '{RequiredSteps[i]}' is recorded before '{RequiredSteps[i - 1]}'.");
var durations = new List<(string, TimeSpan)>();
for (var i = 1; i < RequiredSteps.Length; i++)
durations.Add((RequiredSteps[i], byStep[RequiredSteps[i]] - byStep[RequiredSteps[i - 1]]));
return new DrillSummary(
DetectToDeclare: byStep["Declared"] - byStep["Detected"],
DeclareToRecovered: byStep["Verified"] - byStep["Declared"],
MeasuredRto: byStep["Verified"] - byStep["FaultInjected"],
MeasuredRpo: measuredRpo,
StepDurations: durations);
}
public static string ToMarkdown(DrillSummary s, Target regionTarget)
{
var verdict = s.MeasuredRto <= regionTarget.Rto && s.MeasuredRpo <= regionTarget.Rpo ? "MET" : "NOT MET";
var lines = new List<string>
{
"| Step | Duration |",
"|---|---|"
};
lines.AddRange(s.StepDurations.Select(d => $"| {d.Step} | {d.Duration.TotalSeconds:F0} s |"));
lines.Add("");
lines.Add($"Measured RTO {s.MeasuredRto.TotalMinutes:F1} min vs target {regionTarget.Rto.TotalMinutes:F0} min; " +
$"measured RPO {s.MeasuredRpo.TotalMinutes:F1} min vs target {regionTarget.Rpo.TotalMinutes:F0} min: **{verdict}**.");
return string.Join(Environment.NewLine, lines);
}
}A console entry point that uses fictional timestamps from a tabletop game day:
using Harbourline.Ha.GameDay;
var t0 = new DateTimeOffset(2026, 11, 18, 11, 0, 0, TimeSpan.Zero);
var events = new[]
{
new DrillEvent(t0, "FaultInjected", "Facilitator", "Primary Region database unreachable"),
new DrillEvent(t0.AddMinutes(3), "Detected", "On-call", "Alarm: payment error rate"),
new DrillEvent(t0.AddMinutes(9), "Declared", "Incident commander", "Two approvers recorded"),
new DrillEvent(t0.AddMinutes(17), "Promoted", "DBA", "failover-global-cluster done"),
new DrillEvent(t0.AddMinutes(24), "Repointed", "App engineer","Writer endpoint verified"),
new DrillEvent(t0.AddMinutes(31), "Verified", "QA", "Synthetic payment succeeded")
};
var summary = DrillTimeline.Analyse(events, measuredRpo: TimeSpan.FromMinutes(2));
var regionTarget = new Target(Rto: TimeSpan.FromMinutes(30), Rpo: TimeSpan.FromMinutes(5));
Console.WriteLine(DrillTimeline.ToMarkdown(summary, regionTarget));
var policy = new Dictionary<Disruption, Target>
{
[Disruption.Region] = regionTarget,
[Disruption.AvailabilityZone] = new(TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(1))
};
var observed = new Dictionary<Disruption, Observed>
{
[Disruption.Region] = new(summary.MeasuredRto, summary.MeasuredRpo, "game day 2026-11-18")
// No observation for the AZ disruption: the checker reports "no evidence".
};
foreach (var f in PolicyChecker.Check(policy, observed))
Console.WriteLine($"{f.Disruption,-18} met={f.Met,-5} rtoGap={f.RtoGap} source={f.Source}");Sample output:
| Step | Duration |
|---|---|
| Detected | 180 s |
| Declared | 360 s |
| Promoted | 480 s |
| Repointed | 420 s |
| Verified | 420 s |
Measured RTO 31.0 min vs target 30 min; measured RPO 2.0 min vs target 5 min: **NOT MET**.
Region met=False rtoGap=00:01:00 source=game day 2026-11-18
AvailabilityZone met=False rtoGap=10675199.02:48:05.4775807 source=no evidenceThe tabletop shows a typical finding: the technical steps are quick, but six minutes went on the decision to declare a disaster. The improvement is organisational (pre-agreed declaration criteria and delegated authority), not technical.
Code explained line by line
Disruptionmirrors the four Resilience Hub disruption types, so the same vocabulary is used in the policy, the assessment and the drill.PolicyChecker.Checkcompares each target with an observation. Gaps areobserved minus target: a positive RTO gap means slower than allowed.- No evidence counts as not met. This is the key governance idea: an untested objective is not a met objective.
DrillEventis one timestamped line in the scribe's log: when, which step, who, and a note.Analyserequires six steps in order (FaultInjectedtoVerified) and throws if any is missing or recorded out of order, because a timeline with gaps is not evidence.MeasuredRtoruns from fault injection to verified recovery, not to the moment the database was promoted. Customers care about the latter, which is a more honest RTO.StepDurationsshows where time goes. The analyser separates decision time (Declared) from technical time (Promoted,Repointed).ToMarkdownmakes a table that can be pasted straight into the debrief document and the evidence pack.- In
Program.csthe timestamps are fictional. The AZ disruption has no observation, so its finding says "no evidence", which is exactly what an auditor should see until you run an AZ game day.
Security considerations
⚠ Learning shortcut — not for production
The lab imports one stack in a sandbox with a broad read role and uses a tabletop instead of a live failover. In production, give the assessment role read-only access scoped to the application's stacks, run live game days only with approved change records and rollback plans, restrict who can run the break-glass actions, and send drill logs to protected storage.
- Least privilege. Resilience Hub needs permissions to read the resources of the application (the service uses a role you provide or a service-linked role ⚠ VERIFY). Do not use administrator access.
- Game-day safety. Define the abort criteria and a person with authority to call "stop". Do not run a live Region failover of customer-facing payments without business approval, customer communication and a way back.
- Data in the drill. Use test or synthetic data in rehearsals; restoring production data into a test environment brings production controls with it.
- Evidence integrity. Keep timelines in an append-only store, with names and times from system logs (CloudTrail, chat timestamps) rather than recalled from memory.
- Blameless culture. Debriefs focus on system and process weaknesses. If people fear blame they will hide issues, and the programme loses value.
- Regulatory context. Scenarios should come from your firm's operational-resilience mapping of important business services and severe but plausible events. Confirm with your resilience and compliance functions.
Banking use case
Harbourline runs a game-day calendar per important business service:
| Service | Scenario | Type | Pass criteria |
|---|---|---|---|
| Faster Payments | Primary database unavailable | Live in test, tabletop in prod | RTO 30 min, RPO 5 min |
| Online banking | Loss of one Availability Zone at peak | Live in production off-peak, with FIS | No customer-visible errors above 0.1% |
| Card authorisation | Bad deployment with a corrupt config | Live in staging, rollback drill | Rollback in 10 min |
| Customer notifications | Third-party SMS provider down | Local fault injection and tabletop | Queue and retry; no loss |
Each exercise follows the same shape: a one-page plan (objectives, scenario, roles, injects, success criteria, abort criteria), a scribe-kept timeline, a debrief, and a tracked list of fixes. Resilience Hub assessments run in the pipeline and fail the build if a component drops from Policy met to Policy breached, which catches design drift between game days.
A game-day plan template:
| Section | Content |
|---|---|
| Objective | What question the exercise answers, for example "Can we fail over the ledger in 30 minutes?" |
| Scenario | The injected event and what participants are told |
| Roles | Incident commander, DBA, application engineer, communications, scribe, facilitator, observer |
| Injects | Timed surprises: a second alarm, an unavailable approver, a partner call |
| Success criteria | RTO, RPO, no unreviewed manual change, communication sent in 15 minutes |
| Abort criteria | Steady-state alarm, customer impact, any participant calls stop |
| Evidence | Timeline, screenshots or logs, measured RTO and RPO, list of findings |
Common mistakes
- Treating "Policy met" as proof. It is an estimate. Always time the real SOP.
- Setting targets nobody can meet and then ignoring the breach. Either improve the design or change the policy with a documented decision; do not leave a permanently red dashboard.
- Tabletop only, forever. Paper drills find process gaps but miss technical ones (permissions, DNS caching, quotas). Move to live exercises in non-production, then production.
- A drill without a scribe. Without timestamps there is no RTO. One person records, with times taken from systems.
- Forgetting the decision time. In real incidents, minutes disappear into "who can approve this?". Include it in the timeline.
- No follow-through. Findings without owners and dates stay findings. Track them to closure and rerun.
- Surprising the team without safeguards. Unannounced drills require careful approval and an abort path; start announced.
- Describing only part of the application. A missing component means the assessment is blind to its weaknesses.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Resource not found in the application | Resource type unsupported or stack not imported | Check the supported-resource list; import the right stack or add the resource mapping |
| Assessment fails | Role lacks permissions to read a resource or an unsupported configuration | Read the failure reason; grant read access scoped to the stack |
| Status Policy breached for Region disruption | No cross-Region backup, replica or standby in the definition | Add the cross-Region mechanism (backup copy, global database), reassess |
| Changes detected appears constantly | Infrastructure changes since the last published version | Publish a new version after changes, or reassess in the pipeline |
| Score does not improve after fixes | Recommendations for alarms, SOPs and tests not implemented | Implement the alarm and SOP recommendations as well as the architecture ones |
| Game-day timeline has gaps | No scribe, or events logged from memory | Assign a scribe; export CloudTrail and chat logs afterwards |
| Failover step takes much longer than the runbook says | Runbook untested; missing permissions; DNS TTL | Fix the runbook, test permissions in advance, lower TTLs |
| FIS experiment recommended by Resilience Hub will not start | Missing experiment role or unsupported action in your Region | Create the role; check Region support ⚠ VERIFY |
Cost and cleanup
Cost: credits. Resilience Hub bills per application per month ⚠ VERIFY the current pricing page; run one assessment, review it, then delete the application the same day. The stacks you describe cost whatever they cost (keep them tiny or reuse a lab stack that you already tear down). A game day in the sandbox costs the resources you exercise plus FIS action-minutes if you use FIS. Free alternative: run the C# policy checker and timeline analyser on a tabletop game day; no AWS account is needed.
# Delete the Resilience Hub application and policy (after exporting the assessment you want to keep).
aws resiliencehub delete-app --app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
--force-delete --region us-east-1 --profile learn-sandbox
aws resiliencehub delete-resiliency-policy \
--policy-arn arn:aws:resiliencehub:us-east-1:111122223333:resiliency-policy/POLICY_ID \
--region us-east-1 --profile learn-sandbox
# If you deployed the policy with CDK or Terraform:
cdk destroy --all --profile learn-sandbox
# terraform destroy
# Verify nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"Exercises
- 🟢 Guided: Write the resilience policy for Harbourline's online banking service (four disruption types, RTO and RPO). Run the C# timeline analyser with the sample events, then change the
Declaredevent to 2 minutes afterDetectedand observe the verdict. - 🟡 Challenge: Extend
DrillTimelineto accept multiple objectives (for example a 10-minute target forDeclared) and write xUnit v3 tests for missing steps, out-of-order steps, the verdict boundary (exactly on target is met) and the markdown output. - 🔴 Live problem: The chief risk officer asks for "proof, by next month, that we can stay within our impact tolerance for the payments service in a Region outage." You have one sandbox, a tabletop-only approval for production and a 20-person on-call rota. Propose the evidence plan: assessments, experiments, drills, what you will measure, what you will not claim, and what decisions you need from the business.
Assignment
Write a game-day pack for one Harbourline service: resilience policy (four disruption types), scenario, roles, injects, success and abort criteria, a timeline template, and the debrief template. Include the C# analyser with unit tests and a filled-in timeline from a tabletop run. Commit it to your portfolio repository.
Solutions
A sensible policy for online banking: Software RTO 10 minutes, RPO 5 minutes; Hardware RTO 5 minutes, RPO 1 minute; AZ RTO 15 minutes, RPO 1 minute; Region RTO 60 minutes, RPO 15 minutes (online banking is less time-critical than payments). With Declared at 2 minutes after Detected, the sample's measured RTO drops from 31 to 27 minutes and the verdict flips to MET. The lesson: a large share of the RTO was the decision, so pre-approved declaration criteria are a cheap, effective improvement.
Add an Objectives list of (Step, MaxDuration) and a method that returns the steps that exceeded their budget. Tests: a timeline missing Promoted throws InvalidOperationException mentioning the step; a timeline where Verified precedes Repointed throws; with MeasuredRto exactly equal to the target, ToMarkdown says MET (the comparison is <=); with one second over, it says NOT MET; a step exceeding its own budget is listed. Use fixed DateTimeOffset values so tests are deterministic. The reasoning is that the boundary conditions are where a "met" verdict could be wrongly awarded.
Knowledge check
Answer: Application (software), cloud infrastructure (hardware), Availability Zone disruption, and Region incident. A policy sets RTO and RPO for each.
Answer: The status is an estimate from configuration. A game day times the real SOPs, exposes permission, DNS and people problems, and produces measured RTO and RPO that can serve as evidence.
Answer: After an assessment, Resilience Hub reports policy drift (a component that met the policy before and no longer does) and resource drift (resources added or removed in the application's inputs). It catches design changes that quietly break resilience.
Answer: An objective that has never been measured cannot be claimed as achieved. Treating it as unmet pushes teams to test, which is the point of resilience governance.
Answer: It examines decisions in the context available at the time and focuses on system and process improvements, not individuals. People who fear blame hide problems, so the organisation loses the learning.
Interview questions
- How would you prove to a regulator that you can meet an impact tolerance? Define the tolerance for the service, assess the architecture against it (Resilience Hub or equivalent), run severe but plausible scenarios in controlled exercises with timed evidence, fix the findings and repeat. Present measured results, not assertions, and state the limits of what was tested.
- What goes into a game-day plan? Objective, scenario, roles, injects, success and abort criteria, communications, evidence collection, and a debrief with an action tracker.
- What does Resilience Hub not tell you? It estimates from configuration; it does not time your humans, test third-party dependencies it cannot see, or replace a rehearsal.
- How do you stop resilience work becoming a once-a-year exercise? Put assessments into the pipeline, set drift alerts, schedule rotating game days and track findings to closure.
Certification practice
A company wants an automated assessment that compares its application's configuration with stated RTO and RPO targets and recommends improvements, including alarms and recovery procedures. Which service should it use?
A. AWS Trusted Advisor. B. AWS Resilience Hub. C. Amazon Inspector. D. AWS Config conformance packs.
Answer: B. Resilience Hub takes a resilience policy with RTO and RPO per disruption type and assesses the application, giving estimated workload RTO and RPO and recommendations. A gives general checks, not policy-based assessment. C finds vulnerabilities. D checks configuration rules, not recovery estimates.
An assessment reports Policy breached for Region disruption. The database has Multi-AZ but no cross-Region copy. What is the most direct fix?
A. Add more instances in the same Region. B. Add a cross-Region recovery mechanism, such as an Aurora Global Database secondary or cross-Region backup copies, and reassess. C. Raise the RTO in the policy without changing the design. D. Enable detailed monitoring.
Answer: B. Multi-AZ does not protect against Region loss; a cross-Region mechanism is needed. A does not change the failure domain. C hides the gap unless it is a deliberate, documented business decision. D adds visibility, not recovery.
A team wants to be warned when an infrastructure change makes an application stop meeting its resilience policy. What should they do?
A. Run Resilience Hub assessments on a schedule or in the CI/CD pipeline and act on policy drift notifications. B. Check the AWS Health Dashboard daily. C. Rely on CloudTrail logs. D. Run a game day every two years.
Answer: A. Regular assessments detect policy drift and resource drift. B covers AWS service events, not your configuration. C records changes without evaluating resilience. D is far too infrequent to catch drift.
Completion checklist
- I can explain describe, define, assess, validate and track in Resilience Hub
- I can write a policy with RTO and RPO for the four disruption types
- I ran the timeline analyser on a tabletop and found where the time went
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Resilience Hub estimates whether an application meets RTO and RPO targets for application, infrastructure, AZ and Region disruptions and recommends fixes.
- An estimate is not proof: game days time the real procedures and produce measured RTO and RPO.
- Untested objectives count as unmet.
- Decision time is often the largest part of recovery time: pre-agree declaration criteria.
- Put assessments in the pipeline, drill regularly, run blameless debriefs and close every finding.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Chaos engineering with AWS Fault Injection Service
Plan and run a safe first chaos experiment: write a steady-state hypothesis, build an AWS Fault Injection Service experiment template that stops ECS tasks with a CloudWatch alarm as the stop condition, gate it with a pre-flight check in C#, and rehearse the same idea locally for free.
Well-Architected reviews for a banking API
Run a Well-Architected review of a banking API, focusing on the Reliability pillar and the Financial Services Industry Lens, record risks in the Well-Architected Tool, and turn high-risk items into a tracked improvement plan with C# tooling.