Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

AWS Resilience Hub and game days

Define a resilience policy with RTO and RPO targets, assess an application with AWS Resilience Hub, then run a realistic failover game day and turn the timings into evidence, using a C# policy checker and drill timeline analyser.

advanced⏱ 2 h 30 min🟡 Low costSAA-C03SOA-C03DOP-C02SAP-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.ResilienceHub) · AWS CLI v2 · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Region us-east-1 (DR us-west-2). Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what AWS Resilience Hub does: describe an application, define a resilience policy, run an assessment, act on recommendations, and validate with tests.
  • Write a resilience policy with RTO and RPO targets for the four disruption types: application, cloud infrastructure, Availability Zone and Region.
  • Read an assessment's estimated workload RTO and RPO, its compliance status and its drift reports.
  • Plan and run a game day: scenario, roles, injects, timeline, success criteria and a blameless debrief.
  • Convert a drill's timestamps into measured RTO and RPO and compare them with the policy using C#.

Prerequisites

Why this matters in banking

Harbourline's disaster-recovery document says "RTO 30 minutes, RPO 5 minutes" for the payments service. An internal audit asks three questions: Who decided those numbers? What evidence says the architecture can meet them? When was it last exercised, and what changed afterwards?

Resilience Hub helps with the second question: it estimates, from the real resource configuration, whether the application can meet the policy, and tells you what is missing (no cross-Region backup, a single-AZ database, no alarm for a key metric). Game days answer the third: a rehearsal where people follow the runbook under time pressure and the clock produces the actual RTO. Together they replace "we believe" with "we measured".

Regulators expect firms to set impact tolerances and to test their ability to remain within them using severe but plausible scenarios. This lesson shows engineering methods that produce evidence for that; it does not make a firm compliant, and your risk and compliance teams define the scenarios and approve the tests.

The idea in plain language

Resilience Hub is a checking service. You tell it what your application is made of (CloudFormation stacks, Terraform state, a resource group, an EKS cluster or a myApplications definition) and what you promise (the resilience policy). It looks at the configuration, estimates how quickly and how completely you could recover from each type of disruption, and lists the gaps with recommended fixes, alarms, tests and standard operating procedures (SOPs).

A game day is a rehearsal: you announce a realistic failure, the on-call team responds as in a real incident, and a scribe records every decision and time. The result is a timeline and a list of improvements.

Analogy: Resilience Hub is the building inspector who reads the plans and says "this stairwell is too narrow for 200 people in 4 minutes". The game day is the fire drill where you time the evacuation. You need both: plans can look fine and fail in practice, and a drill cannot find a design flaw nobody thought to exercise.

What problem each tool solves

Resilience Hub concepts

  • Application: a collection of supported resources that Resilience Hub continuously monitors and assesses.
  • Application component (AppComponent): a group of related resources that fail together, such as a primary database and its replica. Components have types, for example compute or database.
  • Resilience policy: RTO and RPO targets for four disruption types:
    • Application (software): infrastructure is healthy but the software misbehaves, for example a bad deployment, a configuration change, data corruption or a failing dependency.
    • Cloud infrastructure (hardware): a component fails locally and is fixed by rebooting or replacing it.
    • Cloud infrastructure AZ disruption: one or more Availability Zones are unavailable.
    • Cloud infrastructure Region incident: one or more Regions are unavailable.
  • Policy tiers: a label such as mission critical, critical, important, core services or non-critical that helps you pick sensible targets ⚠ VERIFY the exact tier names.
  • Assessment: the analysis comparing the application with the policy. It produces an estimated workload RTO and RPO per disruption type and a compliance status: Policy met, Policy breached, Not assessed or Changes detected.
  • Resilience score: a number showing how closely the application follows the recommendations for the policy, alarms, SOPs and tests.
  • Drift detection: reports policy drift (a component that met the policy last time and no longer does) and resource drift (resources added or removed).
  • Recommendations: architecture changes, alarms, SOPs and FIS experiment templates. Some come as deployable code.

Resilience Hub can also help you create and run FIS experiments to verify resilience against these outage types, and tests for SOPs measure the estimated workload RTO and RPO.

What it cannot do

  • It estimates; it does not prove. An estimate from configuration is not a restore. AWS's own wording expects you to maintain and time your recovery SOPs even when the status is Policy met.
  • It only sees what you describe. Resources not in the application definition (for example, a third-party API or an on-premises core system) are outside the assessment.
  • It covers a list of supported resources. Check the supported resources page ⚠ VERIFY for your stack.
  • It costs money. Pricing is per application assessed ⚠ VERIFY the current model; keep sandbox use short.
  • It is not a compliance certificate. A Policy met status is a useful engineering signal, not a regulatory outcome.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Well-Architected Review and Azure Advisor reliability recommendationsAWS Resilience Hub assessmentsResilience Hub compares the resources with explicit RTO and RPO targets you define
Azure Site Recovery recovery plans and test failoverRecovery SOPs and game days; AWS Elastic Disaster Recovery drillsThe "recovery plan" is runbooks plus automation you assemble
Azure Chaos Studio experimentsAWS FIS experiments, recommended by Resilience HubExperiments can be tied to the policy's disruption types
Service Health and Resource HealthAWS Health DashboardHealth events tell you when to start a response; Resilience Hub does not monitor AWS-wide events
Business continuity exercises run by risk teamsGame days run by engineering and operationsIn a bank both exist; game days give the technical evidence for the risk exercise

Architecture

The loop matters more than any single step: assess, fix, rehearse, measure, adjust the policy or the design, assess again. Measured results from the game day should feed back into the policy (maybe the 30-minute RTO is unrealistic) or the design (add warm standby).

Security-boundary view: the assessment role reads resource configuration across the application's stacks, which is sensitive architecture information, so limit it by IAM. Game-day actions use break-glass roles with MFA, run in the DR and production accounts under change approval, and all actions are logged in CloudTrail for the evidence pack.

Step-by-step implementation

Choose the service and set targets with the business

For the payments service: application disruption RTO 10 minutes (rollback), infrastructure 5 minutes (replace a task), AZ 15 minutes, Region 30 minutes with RPO 5 minutes. The targets come from the impact tolerance, not from what is easy.

Create the resilience policy

Define the four disruption types with RTO and RPO in seconds. Keep the policy in IaC or a version-controlled JSON file.

Describe the application

Import resources from the CloudFormation stacks (or a Terraform state file in S3) of the service. Check that every AppComponent has the right resources, including the database and its replicas, and that cross-Region resources appear.

Publish and run an assessment

Publish a version of the application and start an assessment. Wait for it to finish, then read the compliance status and the estimated RTO and RPO.

Triage the recommendations

Group them into must-fix (breaches the policy), should-fix (alarms and SOPs missing) and could-fix. Turn them into tickets with owners.

Plan the game day

Choose a severe but plausible scenario, for example "the primary Region's database is unavailable at 11:00 on a weekday". Write the objectives, success criteria (RTO 30 minutes, RPO 5 minutes), roles, injects, rollback plan and communications. Get approval.

Run it, time it, debrief

Run the scenario, record timestamps for detect, declare, decide, promote, repoint, verify and recover. Hold a blameless debrief within a week, write the improvements and rerun the failed parts.

AWS Console instructions

  • Create a policy: Console → AWS Resilience Hub → Resiliency policies → Create resiliency policy. Pick a tier, or define custom targets. Enter RTO and RPO for Application, Cloud infrastructure, Availability Zone and Region disruptions.
  • Add an application: Applications → Add application → Describe: choose the input source (CloudFormation stack, Terraform state file, resource group, EKS or myApplications), select the resources, then review AppComponents → Define: attach the policy → Publish.
  • Assess: open the application → Assessments → Run resiliency assessment. Review Compliance, the Resilience score, and tabs for Recommendations, Alarms, SOPs and Tests ⚠ VERIFY tab names.
  • Drift: the application dashboard shows drift notifications after each assessment.
  • Delete: open the application → Actions → Delete. Then delete any policy or experiment you created.

AWS CLI

# 1. Create a policy. Keys are AZ, Hardware, Software and Region; values are in seconds.
aws resiliencehub create-resiliency-policy \
  --policy-name hl-m12-payments-policy \
  --tier Critical \
  --policy '{
    "Software": {"rtoInSecs": 600,  "rpoInSecs": 300},
    "Hardware": {"rtoInSecs": 300,  "rpoInSecs": 60},
    "AZ":       {"rtoInSecs": 900,  "rpoInSecs": 60},
    "Region":   {"rtoInSecs": 1800, "rpoInSecs": 300}
  }' \
  --tags Project=cea-lab,Lab=m12-resilience-hub \
  --region us-east-1 --profile learn-sandbox

# 2. Create the application and attach the policy.
aws resiliencehub create-app --name hl-m12-payments \
  --policy-arn arn:aws:resiliencehub:us-east-1:111122223333:resiliency-policy/POLICY_ID \
  --region us-east-1 --profile learn-sandbox

# 3. Import resources from a CloudFormation stack into the draft version.
aws resiliencehub import-resources-to-draft-app-version \
  --app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
  --source-arns arn:aws:cloudformation:us-east-1:111122223333:stack/hl-m12-payments/STACK_ID \
  --region us-east-1 --profile learn-sandbox

# 4. Publish a version, then start an assessment.
aws resiliencehub publish-app-version \
  --app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
  --region us-east-1 --profile learn-sandbox

aws resiliencehub start-app-assessment \
  --app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
  --app-version release --assessment-name hl-m12-first \
  --region us-east-1 --profile learn-sandbox

# 5. Read the result: compliance status and estimated RTO/RPO per disruption type.
aws resiliencehub describe-app-assessment \
  --assessment-arn arn:aws:resiliencehub:us-east-1:111122223333:app-assessment/ASSESSMENT_ID \
  --query "assessment.{Status:assessmentStatus,Compliance:complianceStatus,Score:resiliencyScore.score}" \
  --region us-east-1 --profile learn-sandbox

# 6. List recommendations and alarm recommendations.
aws resiliencehub list-app-component-recommendations \
  --assessment-arn arn:aws:resiliencehub:us-east-1:111122223333:app-assessment/ASSESSMENT_ID \
  --region us-east-1 --profile learn-sandbox

# 7. Cleanup: delete the application and policy.
aws resiliencehub delete-app --app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
  --force-delete --region us-east-1 --profile learn-sandbox
aws resiliencehub delete-resiliency-policy \
  --policy-arn arn:aws:resiliencehub:us-east-1:111122223333:resiliency-policy/POLICY_ID \
  --region us-east-1 --profile learn-sandbox

IDs and the account number are placeholders. Command and parameter names (--source-arns, --force-delete, the tier names and the policy keys) follow the Resilience Hub API at the time of writing, but they have changed between releases; run aws resiliencehub help and check each subcommand before relying on them ⚠ VERIFY.

Infrastructure as Code

infra/cdk/ResiliencePolicyStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.ResilienceHub;
using Constructs;

namespace Harbourline.Ha.Infra;

public sealed class ResiliencePolicyStack : Stack
{
    public ResiliencePolicyStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        // The policy as code: targets are reviewed in a pull request like any other change.
        var policy = new CfnResiliencyPolicy(this, "PaymentsPolicy", new CfnResiliencyPolicyProps
        {
            PolicyName = "hl-m12-payments-policy",
            Tier = "Critical",
            Policy = new Dictionary<string, object>
            {
                ["Software"] = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 600,  RpoInSecs = 300 },
                ["Hardware"] = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 300,  RpoInSecs = 60 },
                ["AZ"]       = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 900,  RpoInSecs = 60 },
                ["Region"]   = new CfnResiliencyPolicy.FailurePolicyProperty { RtoInSecs = 1800, RpoInSecs = 300 }
            },
            Tags = new Dictionary<string, string>
            {
                ["Project"] = "cea-lab", ["Lab"] = "m12-resilience-hub", ["Environment"] = "learn"
            }
        });

        _ = new CfnOutput(this, "PolicyArn", new CfnOutputProps { Value = policy.AttrPolicyArn });
    }
}

Resilience Hub applications can also be declared with CfnApp, but the application template body and resource mappings are detailed JSON; for the lab, importing from a deployed stack with the CLI or console is simpler ⚠ VERIFY CfnResiliencyPolicy property shapes in your CDK version.

Complete code

Two pure C# tools: a policy checker that compares policy targets with measured or estimated values and reports the gap, and a game-day timeline analyser that turns timestamps into measured RTO and RPO.

src/Harbourline.Ha.GameDay/ResiliencePolicy.cs
namespace Harbourline.Ha.GameDay;

public enum Disruption { Software, Hardware, AvailabilityZone, Region }

public sealed record Target(TimeSpan Rto, TimeSpan Rpo);

public sealed record Observed(TimeSpan Rto, TimeSpan Rpo, string Source); // "assessment estimate" or "game day"

public sealed record PolicyFinding(Disruption Disruption, bool Met, TimeSpan RtoGap, TimeSpan RpoGap, string Source);

public static class PolicyChecker
{
    public static IReadOnlyList<PolicyFinding> Check(
        IReadOnlyDictionary<Disruption, Target> policy,
        IReadOnlyDictionary<Disruption, Observed> observed)
    {
        var findings = new List<PolicyFinding>();
        foreach (var (disruption, target) in policy)
        {
            if (!observed.TryGetValue(disruption, out var o))
            {
                // No evidence at all is a failure: untested means not met.
                findings.Add(new PolicyFinding(disruption, false, TimeSpan.MaxValue, TimeSpan.MaxValue, "no evidence"));
                continue;
            }
            var rtoGap = o.Rto - target.Rto;   // positive means slower than the target
            var rpoGap = o.Rpo - target.Rpo;   // positive means more data loss than the target
            findings.Add(new PolicyFinding(disruption,
                Met: rtoGap <= TimeSpan.Zero && rpoGap <= TimeSpan.Zero,
                RtoGap: rtoGap, RpoGap: rpoGap, Source: o.Source));
        }
        return findings;
    }
}
src/Harbourline.Ha.GameDay/DrillTimeline.cs
namespace Harbourline.Ha.GameDay;

public sealed record DrillEvent(DateTimeOffset At, string Step, string Who, string Note = "");

public sealed record DrillSummary(
    TimeSpan DetectToDeclare,
    TimeSpan DeclareToRecovered,
    TimeSpan MeasuredRto,        // from incident start to service verified
    TimeSpan MeasuredRpo,        // data-loss window reported by the database team
    IReadOnlyList<(string Step, TimeSpan Duration)> StepDurations);

public static class DrillTimeline
{
    // Steps in required order; the analyser refuses a timeline that skips or reorders them.
    public static readonly string[] RequiredSteps =
        ["FaultInjected", "Detected", "Declared", "Promoted", "Repointed", "Verified"];

    public static DrillSummary Analyse(IEnumerable<DrillEvent> events, TimeSpan measuredRpo)
    {
        var ordered = events.OrderBy(e => e.At).ToList();
        var byStep = ordered.GroupBy(e => e.Step).ToDictionary(g => g.Key, g => g.First().At);

        foreach (var step in RequiredSteps)
            if (!byStep.ContainsKey(step))
                throw new InvalidOperationException($"Timeline is missing the required step '{step}'.");

        for (var i = 1; i < RequiredSteps.Length; i++)
            if (byStep[RequiredSteps[i]] < byStep[RequiredSteps[i - 1]])
                throw new InvalidOperationException(
                    $"Step '{RequiredSteps[i]}' is recorded before '{RequiredSteps[i - 1]}'.");

        var durations = new List<(string, TimeSpan)>();
        for (var i = 1; i < RequiredSteps.Length; i++)
            durations.Add((RequiredSteps[i], byStep[RequiredSteps[i]] - byStep[RequiredSteps[i - 1]]));

        return new DrillSummary(
            DetectToDeclare: byStep["Declared"] - byStep["Detected"],
            DeclareToRecovered: byStep["Verified"] - byStep["Declared"],
            MeasuredRto: byStep["Verified"] - byStep["FaultInjected"],
            MeasuredRpo: measuredRpo,
            StepDurations: durations);
    }

    public static string ToMarkdown(DrillSummary s, Target regionTarget)
    {
        var verdict = s.MeasuredRto <= regionTarget.Rto && s.MeasuredRpo <= regionTarget.Rpo ? "MET" : "NOT MET";
        var lines = new List<string>
        {
            "| Step | Duration |",
            "|---|---|"
        };
        lines.AddRange(s.StepDurations.Select(d => $"| {d.Step} | {d.Duration.TotalSeconds:F0} s |"));
        lines.Add("");
        lines.Add($"Measured RTO {s.MeasuredRto.TotalMinutes:F1} min vs target {regionTarget.Rto.TotalMinutes:F0} min; " +
                  $"measured RPO {s.MeasuredRpo.TotalMinutes:F1} min vs target {regionTarget.Rpo.TotalMinutes:F0} min: **{verdict}**.");
        return string.Join(Environment.NewLine, lines);
    }
}

A console entry point that uses fictional timestamps from a tabletop game day:

src/Harbourline.Ha.GameDay/Program.cs
using Harbourline.Ha.GameDay;

var t0 = new DateTimeOffset(2026, 11, 18, 11, 0, 0, TimeSpan.Zero);

var events = new[]
{
    new DrillEvent(t0,                         "FaultInjected", "Facilitator", "Primary Region database unreachable"),
    new DrillEvent(t0.AddMinutes(3),           "Detected",      "On-call",     "Alarm: payment error rate"),
    new DrillEvent(t0.AddMinutes(9),           "Declared",      "Incident commander", "Two approvers recorded"),
    new DrillEvent(t0.AddMinutes(17),          "Promoted",      "DBA",         "failover-global-cluster done"),
    new DrillEvent(t0.AddMinutes(24),          "Repointed",     "App engineer","Writer endpoint verified"),
    new DrillEvent(t0.AddMinutes(31),          "Verified",      "QA",          "Synthetic payment succeeded")
};

var summary = DrillTimeline.Analyse(events, measuredRpo: TimeSpan.FromMinutes(2));
var regionTarget = new Target(Rto: TimeSpan.FromMinutes(30), Rpo: TimeSpan.FromMinutes(5));
Console.WriteLine(DrillTimeline.ToMarkdown(summary, regionTarget));

var policy = new Dictionary<Disruption, Target>
{
    [Disruption.Region] = regionTarget,
    [Disruption.AvailabilityZone] = new(TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(1))
};
var observed = new Dictionary<Disruption, Observed>
{
    [Disruption.Region] = new(summary.MeasuredRto, summary.MeasuredRpo, "game day 2026-11-18")
    // No observation for the AZ disruption: the checker reports "no evidence".
};

foreach (var f in PolicyChecker.Check(policy, observed))
    Console.WriteLine($"{f.Disruption,-18} met={f.Met,-5} rtoGap={f.RtoGap} source={f.Source}");

Sample output:

| Step | Duration |
|---|---|
| Detected | 180 s |
| Declared | 360 s |
| Promoted | 480 s |
| Repointed | 420 s |
| Verified | 420 s |

Measured RTO 31.0 min vs target 30 min; measured RPO 2.0 min vs target 5 min: **NOT MET**.
Region             met=False rtoGap=00:01:00 source=game day 2026-11-18
AvailabilityZone   met=False rtoGap=10675199.02:48:05.4775807 source=no evidence

The tabletop shows a typical finding: the technical steps are quick, but six minutes went on the decision to declare a disaster. The improvement is organisational (pre-agreed declaration criteria and delegated authority), not technical.

Code explained line by line

  1. Disruption mirrors the four Resilience Hub disruption types, so the same vocabulary is used in the policy, the assessment and the drill.
  2. PolicyChecker.Check compares each target with an observation. Gaps are observed minus target: a positive RTO gap means slower than allowed.
  3. No evidence counts as not met. This is the key governance idea: an untested objective is not a met objective.
  4. DrillEvent is one timestamped line in the scribe's log: when, which step, who, and a note.
  5. Analyse requires six steps in order (FaultInjected to Verified) and throws if any is missing or recorded out of order, because a timeline with gaps is not evidence.
  6. MeasuredRto runs from fault injection to verified recovery, not to the moment the database was promoted. Customers care about the latter, which is a more honest RTO.
  7. StepDurations shows where time goes. The analyser separates decision time (Declared) from technical time (Promoted, Repointed).
  8. ToMarkdown makes a table that can be pasted straight into the debrief document and the evidence pack.
  9. In Program.cs the timestamps are fictional. The AZ disruption has no observation, so its finding says "no evidence", which is exactly what an auditor should see until you run an AZ game day.

Security considerations

⚠ Learning shortcut — not for production

The lab imports one stack in a sandbox with a broad read role and uses a tabletop instead of a live failover. In production, give the assessment role read-only access scoped to the application's stacks, run live game days only with approved change records and rollback plans, restrict who can run the break-glass actions, and send drill logs to protected storage.

  • Least privilege. Resilience Hub needs permissions to read the resources of the application (the service uses a role you provide or a service-linked role ⚠ VERIFY). Do not use administrator access.
  • Game-day safety. Define the abort criteria and a person with authority to call "stop". Do not run a live Region failover of customer-facing payments without business approval, customer communication and a way back.
  • Data in the drill. Use test or synthetic data in rehearsals; restoring production data into a test environment brings production controls with it.
  • Evidence integrity. Keep timelines in an append-only store, with names and times from system logs (CloudTrail, chat timestamps) rather than recalled from memory.
  • Blameless culture. Debriefs focus on system and process weaknesses. If people fear blame they will hide issues, and the programme loses value.
  • Regulatory context. Scenarios should come from your firm's operational-resilience mapping of important business services and severe but plausible events. Confirm with your resilience and compliance functions.

Banking use case

Harbourline runs a game-day calendar per important business service:

ServiceScenarioTypePass criteria
Faster PaymentsPrimary database unavailableLive in test, tabletop in prodRTO 30 min, RPO 5 min
Online bankingLoss of one Availability Zone at peakLive in production off-peak, with FISNo customer-visible errors above 0.1%
Card authorisationBad deployment with a corrupt configLive in staging, rollback drillRollback in 10 min
Customer notificationsThird-party SMS provider downLocal fault injection and tabletopQueue and retry; no loss

Each exercise follows the same shape: a one-page plan (objectives, scenario, roles, injects, success criteria, abort criteria), a scribe-kept timeline, a debrief, and a tracked list of fixes. Resilience Hub assessments run in the pipeline and fail the build if a component drops from Policy met to Policy breached, which catches design drift between game days.

A game-day plan template:

SectionContent
ObjectiveWhat question the exercise answers, for example "Can we fail over the ledger in 30 minutes?"
ScenarioThe injected event and what participants are told
RolesIncident commander, DBA, application engineer, communications, scribe, facilitator, observer
InjectsTimed surprises: a second alarm, an unavailable approver, a partner call
Success criteriaRTO, RPO, no unreviewed manual change, communication sent in 15 minutes
Abort criteriaSteady-state alarm, customer impact, any participant calls stop
EvidenceTimeline, screenshots or logs, measured RTO and RPO, list of findings

Common mistakes

  • Treating "Policy met" as proof. It is an estimate. Always time the real SOP.
  • Setting targets nobody can meet and then ignoring the breach. Either improve the design or change the policy with a documented decision; do not leave a permanently red dashboard.
  • Tabletop only, forever. Paper drills find process gaps but miss technical ones (permissions, DNS caching, quotas). Move to live exercises in non-production, then production.
  • A drill without a scribe. Without timestamps there is no RTO. One person records, with times taken from systems.
  • Forgetting the decision time. In real incidents, minutes disappear into "who can approve this?". Include it in the timeline.
  • No follow-through. Findings without owners and dates stay findings. Track them to closure and rerun.
  • Surprising the team without safeguards. Unannounced drills require careful approval and an abort path; start announced.
  • Describing only part of the application. A missing component means the assessment is blind to its weaknesses.

Troubleshooting

SymptomLikely causeFix
Resource not found in the applicationResource type unsupported or stack not importedCheck the supported-resource list; import the right stack or add the resource mapping
Assessment failsRole lacks permissions to read a resource or an unsupported configurationRead the failure reason; grant read access scoped to the stack
Status Policy breached for Region disruptionNo cross-Region backup, replica or standby in the definitionAdd the cross-Region mechanism (backup copy, global database), reassess
Changes detected appears constantlyInfrastructure changes since the last published versionPublish a new version after changes, or reassess in the pipeline
Score does not improve after fixesRecommendations for alarms, SOPs and tests not implementedImplement the alarm and SOP recommendations as well as the architecture ones
Game-day timeline has gapsNo scribe, or events logged from memoryAssign a scribe; export CloudTrail and chat logs afterwards
Failover step takes much longer than the runbook saysRunbook untested; missing permissions; DNS TTLFix the runbook, test permissions in advance, lower TTLs
FIS experiment recommended by Resilience Hub will not startMissing experiment role or unsupported action in your RegionCreate the role; check Region support ⚠ VERIFY

Cost and cleanup

Cost: credits. Resilience Hub bills per application per month ⚠ VERIFY the current pricing page; run one assessment, review it, then delete the application the same day. The stacks you describe cost whatever they cost (keep them tiny or reuse a lab stack that you already tear down). A game day in the sandbox costs the resources you exercise plus FIS action-minutes if you use FIS. Free alternative: run the C# policy checker and timeline analyser on a tabletop game day; no AWS account is needed.

# Delete the Resilience Hub application and policy (after exporting the assessment you want to keep).
aws resiliencehub delete-app --app-arn arn:aws:resiliencehub:us-east-1:111122223333:app/APP_ID \
  --force-delete --region us-east-1 --profile learn-sandbox
aws resiliencehub delete-resiliency-policy \
  --policy-arn arn:aws:resiliencehub:us-east-1:111122223333:resiliency-policy/POLICY_ID \
  --region us-east-1 --profile learn-sandbox

# If you deployed the policy with CDK or Terraform:
cdk destroy --all --profile learn-sandbox
# terraform destroy

# Verify nothing tagged Project=cea-lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • 🟢 Guided: Write the resilience policy for Harbourline's online banking service (four disruption types, RTO and RPO). Run the C# timeline analyser with the sample events, then change the Declared event to 2 minutes after Detected and observe the verdict.
  • 🟡 Challenge: Extend DrillTimeline to accept multiple objectives (for example a 10-minute target for Declared) and write xUnit v3 tests for missing steps, out-of-order steps, the verdict boundary (exactly on target is met) and the markdown output.
  • 🔴 Live problem: The chief risk officer asks for "proof, by next month, that we can stay within our impact tolerance for the payments service in a Region outage." You have one sandbox, a tabletop-only approval for production and a 20-person on-call rota. Propose the evidence plan: assessments, experiments, drills, what you will measure, what you will not claim, and what decisions you need from the business.

Assignment

Write a game-day pack for one Harbourline service: resilience policy (four disruption types), scenario, roles, injects, success and abort criteria, a timeline template, and the debrief template. Include the C# analyser with unit tests and a filled-in timeline from a tabletop run. Commit it to your portfolio repository.

Solutions

Knowledge check

Interview questions

  • How would you prove to a regulator that you can meet an impact tolerance? Define the tolerance for the service, assess the architecture against it (Resilience Hub or equivalent), run severe but plausible scenarios in controlled exercises with timed evidence, fix the findings and repeat. Present measured results, not assertions, and state the limits of what was tested.
  • What goes into a game-day plan? Objective, scenario, roles, injects, success and abort criteria, communications, evidence collection, and a debrief with an action tracker.
  • What does Resilience Hub not tell you? It estimates from configuration; it does not time your humans, test third-party dependencies it cannot see, or replace a rehearsal.
  • How do you stop resilience work becoming a once-a-year exercise? Put assessments into the pipeline, set drift alerts, schedule rotating game days and track findings to closure.

Certification practice

Completion checklist

  • I can explain describe, define, assess, validate and track in Resilience Hub
  • I can write a policy with RTO and RPO for the four disruption types
  • I ran the timeline analyser on a tabletop and found where the time went
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Resilience Hub estimates whether an application meets RTO and RPO targets for application, infrastructure, AZ and Region disruptions and recommends fixes.
  • An estimate is not proof: game days time the real procedures and produce measured RTO and RPO.
  • Untested objectives count as unmet.
  • Decision time is often the largest part of recovery time: pre-agree declaration criteria.
  • Put assessments in the pipeline, drill regularly, run blameless debriefs and close every finding.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page