Operational resilience and third-party risk
Understand what UK and EU regulators expect when a bank runs important business services on AWS — important business services, impact tolerances, outsourcing and third-party risk, the Critical Third Parties regime, DORA, exit plans and concentration risk — and turn those expectations into architecture, chaos experiments and evidence. Concepts only.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Read this first
This lesson explains regulatory concepts so engineers can design and evidence resilient services. It is not regulatory or legal advice and doesn't make any service "compliant". Harbourline's operational resilience, outsourcing and risk functions, its legal team and its board own the interpretations, impact tolerances, materiality assessments and notifications described here.
Learning objectives
By the end of this lesson you can:
- Explain UK operational resilience expectations: important business services, impact tolerances, mapping, scenario testing and self-assessment, and why "within tolerance by 31 March 2025" mattered.
- Explain outsourcing and third-party risk management (PRA SS2/21 and the FCA's equivalent rules): materiality, due diligence, contractual rights, business continuity and exit planning.
- Describe the UK Critical Third Parties (CTP) regime and the EU DORA regime, including the designation of AWS entities, and why designation doesn't reduce the bank's own responsibilities.
- Distinguish firm-level concentration risk from sector-level concentration, and weigh multi-AZ, multi-Region and multi-cloud options honestly.
- Turn impact tolerances into RTO/RPO targets and architecture, and test them with AWS Fault Injection Service (FIS), Resilience Hub and Application Recovery Controller (ARC).
- Write a practical, testable exit plan for a workload on AWS.
Prerequisites
- Threat modelling with STRIDE
- Audit logging and evidence collection
- Backup security and key rotation in this module, and Module 12: High availability and DR for the deeper resilience patterns
- Tools: .NET 10 SDK; AWS CLI v2 with
learn-sandboxfor the optional read-only commands
Why this matters in banking
On a Friday afternoon, Harbourline's payments service slows down, then stops. The root cause turns out to be a dependency in one AWS Region having elevated error rates for 90 minutes. Customers can't pay rent; small businesses can't pay staff. The questions that follow are not mostly technical:
- Was "make a payment" identified as an important business service, and what was its impact tolerance? Did the outage exceed it?
- Had the bank mapped the service's dependencies, including AWS services and other third parties, and tested a scenario like this?
- What did the outsourcing contract with the cloud provider say about information, audit and incident cooperation?
- If the bank had to leave the provider, is there a credible exit plan?
- Does the regulator need to be notified of the incident, and how quickly?
Regulators no longer ask only "how do you prevent outages?". They assume disruption will happen and ask: "can you stay within tolerable harm when it does, and can you prove it?" That is what this lesson is about.
The idea in plain language
Operational resilience is the ability to prevent, adapt to, respond to, recover from and learn from operational disruption. The UK framework (from the PRA, FCA and Bank of England, policy published March 2021) asks firms to:
- Identify important business services (IBS) — services provided to external customers or market participants whose disruption could cause intolerable harm to consumers, market integrity, the firm's safety and soundness, or financial stability. For Harbourline: making payments, accessing accounts, cash withdrawal, card authorisation.
- Set impact tolerances — the maximum tolerable level of disruption to each IBS, usually expressed as a maximum duration (for example "payments can be unavailable for at most 4 hours") and sometimes a volume or number of customers.
- Map the people, processes, technology, facilities, information and third parties that deliver each IBS.
- Scenario-test whether the firm can stay within tolerance under severe but plausible scenarios, such as loss of a data centre, a cyber attack, or a third-party failure.
- Self-assess and remediate, with board oversight.
Firms had a transition period and had to be able to remain within impact tolerances by 31 March 2025. Since then, supervision focuses on whether firms really can.
Outsourcing and third-party risk management is the other half. The PRA's supervisory statement SS2/21 (with equivalent FCA rules) sets expectations for arrangements with third parties, especially material outsourcing such as running core services on a cloud provider: assess materiality, do due diligence, include specific contractual rights (access, audit, information, sub-outsourcing, termination), plan business continuity, and maintain exit plans for both stressed exit (the provider fails or the relationship breaks down suddenly) and non-stressed exit (a planned move).
New and changing rules (checked 2026-11-18):
| Regime | What it is | Key dates |
|---|---|---|
| UK Critical Third Parties (CTP) | HM Treasury can designate third parties whose failure could threaten UK financial stability; the PRA, FCA and Bank of England then oversee their services to the sector directly | Rules effective 1 January 2025; first designations announced 10 July 2026, effective 13 July 2026, covering four cloud and technology providers including Amazon Web Services EMEA SARL |
| UK incident and third-party reporting | New rules for reporting operational incidents and maintaining/submitting a register of material third-party arrangements (FCA PS26/2, PRA PS7/26, with an updated SS2/21) | Published 18 March 2026; apply from 18 March 2027 |
| EU DORA (Digital Operational Resilience Act, Regulation (EU) 2022/2554) | ICT risk management, incident reporting, resilience testing, ICT third-party risk (register of information, contract terms, exit strategies) and oversight of critical ICT third-party providers | Applies from 17 January 2025; on 18 November 2025 the European Supervisory Authorities designated 19 critical ICT third-party providers, including Amazon Web Services EMEA SARL |
Harbourline's EU subsidiary is subject to DORA; the UK bank is subject to the UK regime. In both, designation of the provider does not reduce the bank's own responsibilities. Regulators oversee the provider's resilience to the sector; the bank still owns its IBS, tolerances, contracts, testing and exit plans.
Analogy: the bank is an airline. Operational resilience is the plan for keeping passengers moving when something goes wrong — spare aircraft, rebooking rules, a maximum delay you'll tolerate before you put people in hotels. Third-party risk management is how you choose and contract with the airport, the fuel supplier and the catering firm. The CTP regime is the aviation authority inspecting the airport itself, because if the airport fails every airline fails. That inspection doesn't mean your airline can skip its own contingency plans.
Concentration risk
| Level | Question | Who mostly owns it |
|---|---|---|
| Firm-level | How dependent is Harbourline on one provider, Region or service? Could it continue IBS if that provider failed or became unavailable? | The bank (SS2/21, DORA) |
| Sector-level | How dependent is the financial system on a few providers? | Regulators (CTP regime, DORA oversight) |
Concentration isn't automatically bad: one well-run provider with multi-AZ and multi-Region design may be more resilient than three poorly integrated ones. The regulatory expectation is that you understand, manage and can explain it.
What problem the AWS services solve
| AWS capability | Resilience use | Limits and honest notes |
|---|---|---|
| Availability Zones and Regions | Fault isolation: multi-AZ for most failures, multi-Region for Region-wide events | Multi-Region doubles cost and complexity; data residency (previous lesson) limits Region choices — London is the only UK Region |
| AWS Resilience Hub | Define RTO/RPO per application, assess the architecture against them, get recommendations and track drift | Assessments, not guarantees; pricing per application per month ⚠ VERIFY |
| AWS Fault Injection Service (FIS) | Run controlled experiments (stop tasks, disrupt networking, inject API errors, simulate AZ power interruption) with stop conditions | Real faults in real accounts — start in non-production; charged per action-minute ⚠ VERIFY |
| Amazon Application Recovery Controller (ARC) | Zonal shift / zonal autoshift to move traffic away from an impaired AZ; Region switch and routing controls for multi-Region failover | Your application must be built to run in the remaining AZs/Regions |
| AWS Backup (cross-account, cross-Region, Vault Lock) | Recover from data loss, ransomware and logical corruption | Restores must be tested; immutable backups are part of a stressed-exit story too |
| AWS Health | Account-specific notifications of AWS events affecting your resources | Organisational view and API access depend on support plan ⚠ VERIFY |
| AWS Artifact | SOC 1/2, ISO, C5 reports for due diligence on AWS | Evidence of AWS's controls only |
| AWS contracts for financial services | AWS offers contractual terms aimed at regulated financial institutions (for example audit and information rights) ⚠ VERIFY current names and availability with your AWS account team and legal | Negotiated by Legal and Procurement, not engineering |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Availability Zones, paired regions | AWS AZs and Regions | AWS has no automatic "region pairs"; you choose and build the secondary Region |
| Azure Chaos Studio | AWS Fault Injection Service | Both use experiment templates with targets and stop conditions |
| Azure Site Recovery | AWS Elastic Disaster Recovery, AWS Backup, ARC | DR for IaaS servers vs. application-level failover |
| Azure Resiliency (formerly Resiliency in Azure Advisor / BCDR guidance) | AWS Resilience Hub | Resilience Hub assesses against RTO/RPO you define per app |
| Microsoft Financial Services Amendment | AWS's financial services contract terms (⚠ VERIFY name) | Both address regulator audit/access rights |
| Microsoft also designated as UK CTP and EU CTPP | AWS EMEA SARL designated in both regimes | Same regulatory position; your obligations are unchanged |
Architecture
From important business service to architecture, tests and evidence:
Mapping view (dependencies for "make a payment")
| Component | Provider | Failure scenario | Mitigation | Tested by |
|---|---|---|---|---|
| Payments API (Fargate) | AWS | AZ impairment | 3 AZs, ARC zonal autoshift | FIS stop-task in one AZ; zonal shift drill |
| Ledger (Aurora PostgreSQL) | AWS | Writer failure, AZ loss | Multi-AZ cluster; failover | FIS / manual failover test |
| Ledger | AWS | Logical corruption, ransomware | Point-in-time restore; Vault Lock backups in another account | Quarterly restore test |
| Whole Region | AWS | Region-wide service impairment | Warm standby in a second Region (if approved), manual failover with ARC Region switch | Annual failover exercise |
| Faster Payments gateway | Scheme / connectivity provider | Gateway outage | Queue and retry; customer messaging; scheme contingency | Tabletop exercise |
| Sanctions screening data | External vendor | Feed stale or unavailable | Cache last good list with age limits; manual review path | Tabletop + simulation |
| Identity provider | Internal / vendor | Staff can't sign in during incident | Break-glass roles with tested procedure | Break-glass drill |
Step-by-step implementation
Agree the IBS and impact tolerances with the business
Engineers don't choose tolerances, but they must understand them. For each IBS, get the tolerance in writing, and translate it into technical targets with the service owner: a 4-hour tolerance for payments might become an RTO of 1 hour for the API (leaving time for detection and decision), an RPO of zero for the ledger (no lost postings) and a recovery-order plan.
Map dependencies, including AWS services and third parties
List every component, which AWS services it uses (and in which Regions), and every external party. Include the "invisible" ones: DNS, certificates, identity provider, CI/CD (you may need to deploy a fix during an incident), monitoring, and the people and runbooks. The mapping feeds the third-party register you'll need for regulatory reporting.
Design for the tolerance, not for perfection
Use multi-AZ everywhere for IBS. Decide multi-Region per IBS based on the tolerance, cost and residency constraints. Make components degrade gracefully (accept payments into a durable queue even if the scheme gateway is down). Keep immutable, cross-account backups.
Test with severe but plausible scenarios
Run FIS experiments in pre-production first (stop tasks in one AZ, inject latency to the database, simulate an AZ power interruption scenario), then carefully in production with stop conditions. Run ARC zonal shift drills. Do restore tests. Run tabletop exercises for scenarios you can't simulate (provider insolvency, a major cyber attack). Record results against the tolerance.
Assess the third party and plan exit
With procurement and risk: materiality assessment, due diligence (Artifact reports, financial standing, sub-processors), contract terms, and an exit plan. Engineering's part of the exit plan is the technical runbook: what data you'd extract and how, what you'd rebuild elsewhere, how long it would take, and what you'd do in the meantime.
Self-assess, remediate, repeat
Feed test results, Resilience Hub assessments and incident lessons into the annual self-assessment. Track vulnerabilities (things that would breach tolerance) as remediation items with owners and dates.
Exit planning on AWS
Exit plans are the part engineers most often hand-wave. A credible plan answers concrete questions:
| Question | Stressed exit (sudden) | Non-stressed exit (planned) |
|---|---|---|
| Trigger | Provider failure, sanctions, legal prohibition, contract breakdown, prolonged severe outage | Commercial or strategic decision |
| Timeframe | Days to weeks; may need to run on a minimal service | Months to years |
| What keeps IBS running meanwhile? | Manual workarounds, alternative channels, reduced service, a minimal deployment elsewhere | Parallel running |
| Data | Regular exports to storage you control outside the provider (encrypted, in open formats), inventory of where data lives | Bulk migration with reconciliation |
| Keys | Can you decrypt your exports without the provider's KMS? (Bring-your-own key material or exports encrypted with keys you hold elsewhere) | Re-encrypt during migration |
| Code and infrastructure | Containers, open-source databases and Terraform/OpenTofu reduce lock-in; provider-specific services need substitutes identified | Re-platform progressively |
| People | Who does what, with what access, documented and rehearsed | Programme team |
Engineering techniques that make exit more credible without abandoning managed services:
- Portable compute — containers (Fargate/ECS/EKS) and .NET apps that run anywhere; keep provider SDK calls behind interfaces (as this platform's code does with
IAmazon…abstractions). - Portable data — PostgreSQL-compatible databases, Parquet/CSV exports, documented schemas. Regularly export critical data (ledger, customer records) to a location the bank controls, encrypted with keys it can use without AWS.
- Infrastructure as code — know which resources exist and how they're configured; identify equivalents for each.
- Rehearse — at least a desk-based walk-through annually, and a technical proof (restore last night's export into a non-AWS environment) for the most important data.
- Data transfer costs — AWS announced in 2024 that customers moving data out of AWS can request waived data-transfer-out charges for the move ⚠ VERIFY current terms with AWS.
AWS Console instructions
- Resilience Hub: Console → AWS Resilience Hub → Applications → Add application. You can describe an app from a CloudFormation stack, Terraform state file, resource group or tags. Define a resiliency policy (RTO/RPO per disruption type: application, infrastructure, AZ, Region), then Assess. Review in-sandbox only if your app exists; otherwise read the workflow.
- FIS: Console → AWS FIS → Experiment templates → Create experiment template. Choose Actions (for example
aws:ecs:stop-task), Targets (selected by tag), and a Stop condition (a CloudWatch alarm). Note the IAM role FIS needs. Create the template; don't start it until you've read the safety notes below. - ARC zonal shift: Console → Application Recovery Controller → Zonal shift. See which load balancers are eligible, and how to start a zonal shift or enable zonal autoshift with practice runs.
- AWS Health: Console → AWS Health Dashboard → Your account health — open and scheduled events affecting your resources.
- AWS Artifact: download SOC 2 Type 2 and ISO 27001 reports for the third-party due diligence file.
AWS CLI
# 1. Resilience Hub: list apps and their latest assessed resiliency score (if any exist).
aws resiliencehub list-apps --query "appSummaries[].{name:name,score:resiliencyScore,compliance:complianceStatus}" \
--output table --profile learn-sandbox
# 2. FIS: list experiment templates and past experiments.
aws fis list-experiment-templates --query "experimentTemplates[].{id:id,description:description}" --output table --profile learn-sandbox
aws fis list-experiments --query "experiments[].{id:id,state:state.status,template:experimentTemplateId}" --output table --profile learn-sandbox
# 3. FIS: see which actions are available (e.g. ECS, EC2, RDS, network, API fault injection).
aws fis list-actions --query "actions[?starts_with(id, 'aws:ecs')].id" --output text --profile learn-sandbox
# 4. ARC zonal shift: resources that can be shifted, and any active shifts.
aws arc-zonal-shift list-managed-resources --profile learn-sandbox
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile learn-sandbox
# 5. AWS Health events affecting the account (the Health API requires a Business, Enterprise On-Ramp
# or Enterprise support plan; otherwise use the console dashboard).
aws health describe-events --region us-east-1 \
--filter eventStatusCodes=open,upcoming --profile learn-sandbox
# 6. Which Regions hold your resources? A quick concentration check for the account.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query "ResourceTagMappingList[].ResourceARN" --output text --profile learn-sandbox | tr '\t' '\n' | cut -d: -f4 | sort | uniq -cInfrastructure as Code
An FIS experiment template that stops one Payments API task (selected by tag) to prove the service stays within its error budget when capacity is lost, with a CloudWatch alarm as the stop condition. The FIS role is narrowly scoped.
using System.Collections.Generic;
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.FIS;
using Amazon.CDK.AWS.IAM;
using Constructs;
namespace Harbourline.Payments.Infra;
public sealed class ResilienceExperimentStack : Stack
{
public ResilienceExperimentStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
// Stop condition: if the API's 5xx rate breaches the SLO, FIS halts the experiment.
var errorAlarm = new Alarm(this, "Api5xxAlarm", new AlarmProps
{
AlarmName = "hl-m05-payments-5xx-during-experiment",
Metric = new Metric(new MetricProps
{
Namespace = "AWS/ApplicationELB",
MetricName = "HTTPCode_Target_5XX_Count",
DimensionsMap = new Dictionary<string, string> { ["LoadBalancer"] = "app/hl-m05-payments/0123456789abcdef" },
Statistic = "Sum",
Period = Duration.Minutes(1),
}),
Threshold = 10,
EvaluationPeriods = 1,
TreatMissingData = TreatMissingData.NOT_BREACHING,
});
var fisRole = new Role(this, "FisRole", new RoleProps
{
RoleName = "hl-m05-fis-experiment",
AssumedBy = new ServicePrincipal("fis.amazonaws.com"),
});
fisRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = new[] { "ecs:StopTask", "ecs:DescribeTasks", "ecs:ListTasks", "ecs:DescribeClusters" },
Resources = new[] { "*" },
Conditions = new Dictionary<string, object>
{
["StringEquals"] = new Dictionary<string, object> { ["aws:ResourceTag/Lab"] = "m05-resilience" },
},
}));
_ = new CfnExperimentTemplate(this, "StopOnePaymentsTask", new CfnExperimentTemplateProps
{
Description = "Stop one Payments API task; service must stay within its 5xx budget",
RoleArn = fisRole.RoleArn,
Targets = new Dictionary<string, object>
{
["paymentsTasks"] = new CfnExperimentTemplate.ExperimentTemplateTargetProperty
{
ResourceType = "aws:ecs:task",
ResourceTags = new Dictionary<string, string> { ["Lab"] = "m05-resilience" },
SelectionMode = "COUNT(1)",
},
},
Actions = new Dictionary<string, object>
{
["stopTask"] = new CfnExperimentTemplate.ExperimentTemplateActionProperty
{
ActionId = "aws:ecs:stop-task",
Targets = new Dictionary<string, string> { ["Tasks"] = "paymentsTasks" },
},
},
StopConditions = new object[]
{
new CfnExperimentTemplate.ExperimentTemplateStopConditionProperty
{
Source = "aws:cloudwatch:alarm",
Value = errorAlarm.AlarmArn,
},
},
Tags = new Dictionary<string, string>
{
["Project"] = "cea-lab", ["Lab"] = "m05-resilience", ["Environment"] = "learn", ["Owner"] = "you",
},
});
}
}ECS tasks carry tags only if the service propagates them (PropagateTags = SERVICE or TASK_DEFINITION). Check the target's tag-based selection for aws:ecs:task in the FIS documentation ⚠ VERIFY; FIS also supports selecting ECS tasks by cluster and service parameters.
Creating the template, role and alarm is effectively free (one standard alarm is within the CloudWatch free tier). Running the experiment costs FIS action-minutes and requires a running service; that's optional.
Complete code
A resilience register as code: each IBS has an impact tolerance and a dependency map, and the evaluator checks a scenario (for example "loss of AZ eu-west-2a" or "sanctions vendor unavailable") against the tolerance, using each component's recovery time. It turns the mapping spreadsheet into something you can test in the pipeline and use in tabletop exercises.
namespace Harbourline.Resilience;
public enum Provider { Aws, SchemeOperator, ExternalVendor, Internal }
/// <summary>A dependency of an important business service.</summary>
/// <param name="FailureDomains">What can take it down, e.g. "az:eu-west-2a", "region:eu-west-2", "vendor:sanctions-data".</param>
/// <param name="SurvivesWith">Failure domains it tolerates thanks to redundancy (e.g. a 3-AZ service survives any single AZ).</param>
/// <param name="RecoveryTime">Time to restore the component if a failure domain it can't survive is lost.</param>
/// <param name="Workaround">A manual or degraded path that keeps the IBS within tolerance, if any.</param>
public sealed record Component(
string Name,
Provider Provider,
IReadOnlySet<string> FailureDomains,
IReadOnlySet<string> SurvivesWith,
TimeSpan RecoveryTime,
string? Workaround = null,
TimeSpan? WorkaroundTime = null);
public sealed record ImportantBusinessService(string Name, TimeSpan ImpactTolerance, IReadOnlyList<Component> Components);
public sealed record Scenario(string Name, IReadOnlySet<string> LostDomains, TimeSpan DetectionAndDecisionTime);
public sealed record ScenarioResult(
string Service, string Scenario, TimeSpan EstimatedDisruption, TimeSpan Tolerance,
bool WithinTolerance, IReadOnlyList<string> CriticalPath);
public static class ScenarioEvaluator
{
public static ScenarioResult Evaluate(ImportantBusinessService ibs, Scenario scenario)
{
var impacts = new List<(string Name, TimeSpan Time)>();
foreach (var c in ibs.Components)
{
var hit = c.FailureDomains.Overlaps(scenario.LostDomains);
if (!hit)
continue;
var lostButSurvivable = scenario.LostDomains.Where(c.FailureDomains.Contains).All(c.SurvivesWith.Contains);
if (lostButSurvivable)
continue;
// Use the faster of full recovery or the documented workaround.
var time = c.WorkaroundTime is { } w && w < c.RecoveryTime ? w : c.RecoveryTime;
var label = c.WorkaroundTime is { } w2 && w2 < c.RecoveryTime ? $"{c.Name} (workaround: {c.Workaround})" : c.Name;
impacts.Add((label, time));
}
// Components recover in parallel; the IBS is restored when the slowest affected component is.
var worst = impacts.Count == 0 ? TimeSpan.Zero : impacts.Max(i => i.Time);
var disruption = impacts.Count == 0 ? TimeSpan.Zero : scenario.DetectionAndDecisionTime + worst;
var critical = impacts.Where(i => i.Time == worst).Select(i => i.Name).ToList();
return new ScenarioResult(ibs.Name, scenario.Name, disruption, ibs.ImpactTolerance,
disruption <= ibs.ImpactTolerance, critical);
}
}using Harbourline.Resilience;
using Xunit;
public class MakeAPaymentTests
{
private static HashSet<string> S(params string[] v) => new(v, StringComparer.Ordinal);
private static readonly ImportantBusinessService MakeAPayment = new(
"Make a payment",
ImpactTolerance: TimeSpan.FromHours(4),
Components:
[
new Component("Payments API (Fargate, 3 AZs)", Provider.Aws,
FailureDomains: S("az:eu-west-2a", "az:eu-west-2b", "az:eu-west-2c", "region:eu-west-2"),
SurvivesWith: S("az:eu-west-2a", "az:eu-west-2b", "az:eu-west-2c"),
RecoveryTime: TimeSpan.FromHours(6)), // rebuild in another Region from IaC + restore
new Component("Ledger (Aurora PostgreSQL, multi-AZ)", Provider.Aws,
FailureDomains: S("az:eu-west-2a", "az:eu-west-2b", "region:eu-west-2"),
SurvivesWith: S("az:eu-west-2a", "az:eu-west-2b"),
RecoveryTime: TimeSpan.FromHours(8)),
new Component("Sanctions screening data feed", Provider.ExternalVendor,
FailureDomains: S("vendor:sanctions-data"),
SurvivesWith: S(),
RecoveryTime: TimeSpan.FromHours(24),
Workaround: "use cached list up to 24h old + manual review of matches",
WorkaroundTime: TimeSpan.FromMinutes(30)),
new Component("Faster Payments gateway", Provider.SchemeOperator,
FailureDomains: S("scheme:fps"),
SurvivesWith: S(),
RecoveryTime: TimeSpan.FromHours(2)),
]);
[Fact]
public void Loss_of_one_AZ_is_absorbed()
{
var r = ScenarioEvaluator.Evaluate(MakeAPayment, new Scenario("AZ a lost", S("az:eu-west-2a"), TimeSpan.FromMinutes(15)));
Assert.True(r.WithinTolerance);
Assert.Equal(TimeSpan.Zero, r.EstimatedDisruption);
}
[Fact]
public void Vendor_outage_stays_within_tolerance_only_because_of_the_workaround()
{
var r = ScenarioEvaluator.Evaluate(MakeAPayment, new Scenario("Sanctions vendor down", S("vendor:sanctions-data"), TimeSpan.FromMinutes(30)));
Assert.True(r.WithinTolerance);
Assert.Contains("workaround", r.CriticalPath.Single());
}
[Fact]
public void Region_loss_breaches_tolerance_without_a_warm_standby()
{
var r = ScenarioEvaluator.Evaluate(MakeAPayment, new Scenario("eu-west-2 impaired", S("region:eu-west-2"), TimeSpan.FromMinutes(30)));
Assert.False(r.WithinTolerance); // 30 min + 8 h > 4 h
Assert.Equal("Ledger (Aurora PostgreSQL, multi-AZ)", r.CriticalPath.Single());
}
}The third test is the point of the exercise: it documents a vulnerability (a Region-wide event would breach the 4-hour tolerance), which goes into the self-assessment with a remediation decision — build a warm standby, accept with board approval, or change the service design.
Code explained line by line
Provider— knowing which components depend on which third party is the start of the third-party register and concentration analysis.FailureDomains— strings such asaz:…,region:…,vendor:…,scheme:…keep the model simple and extensible (addidentity:entra,dns:route53as needed).SurvivesWith— redundancy expressed as data: a three-AZ service survives losing any single AZ. Losing two AZs is a different scenario you can also test.RecoveryTimevsWorkaroundTime— regulators care about the service to customers, not the component; a manual workaround that restores the service in 30 minutes counts, as long as it's documented and rehearsed.ImpactToleranceon the IBS — set by the business and board; engineering tests against it.DetectionAndDecisionTimeon the scenario — real incidents spend time noticing and deciding before recovery starts; leaving it out makes every plan look better than it is.OverlapsthenAll(SurvivesWith.Contains)— a component is unaffected if the lost domains are all ones it's designed to survive.impacts.Max— components recover in parallel (a simplification); the slowest determines restoration. Sequential dependencies can be modelled by adding times.CriticalPath— the components that determine the outcome: where investment changes the answer.- Tests use C# collection expressions and named arguments so the model reads like the mapping document the risk team maintains.
Security considerations
⚠ Learning shortcut — not for production
This lesson's FIS template targets tasks by a lab tag in a sandbox, and the scenario model uses estimated recovery times. In production: run experiments first in pre-production, get change approval for production experiments, use narrow targets and stop conditions, run during staffed hours with the incident team informed, and base recovery times on measured results from tests and real incidents, not estimates. Impact tolerances and the decision to accept a vulnerability belong to the business and board.
- Resilience and security overlap. Ransomware and destructive attacks are among the most severe plausible scenarios; immutable, cross-account backups and tested restores are both security and resilience controls.
- Break-glass access must work when the normal identity provider is down. Test it.
- Exports for exit are sensitive. Data exported for stressed-exit purposes must be encrypted with keys the bank controls, access-restricted and covered by the data inventory and retention rules.
- FIS is a powerful permission. Restrict who can create and start experiments (
fis:StartExperiment) and scope the FIS role with tag conditions. - Don't let multi-cloud weaken security. Two clouds means two identity systems, two sets of guardrails, two logging pipelines. Complexity is itself a risk.
Banking use case
Harbourline's operational resilience team identified eight IBS. For "make a payment" (tolerance: 4 hours), the platform team:
- Mapped 23 components and 6 third parties into the resilience register (similar to the code above), stored in the service repository and reviewed quarterly.
- Ran monthly FIS experiments in pre-production (stop tasks, inject database latency) and quarterly ARC zonal-shift practice runs in production.
- Discovered from the register that a Region-wide event would exceed tolerance because the ledger restore took about 8 hours. The board approved investment in a warm standby in a second Region, subject to the DPO's assessment of the data-residency implications of a non-UK Region (London is the only UK AWS Region). Until it was built, the vulnerability was recorded in the self-assessment with an interim workaround (a manual payments channel for critical corporate payments).
- Wrote a stressed-exit runbook: nightly encrypted exports of the ledger and customer records to storage the bank controls outside AWS, with keys held in the bank's on-premises HSM; container images and IaC mirrored; an annual rehearsal restoring the ledger export into a non-AWS PostgreSQL instance and reconciling balances.
- Supplied the outsourcing team with the AWS service list, Regions, sub-processors and Artifact reports for the material-outsourcing notification and the third-party register that the new UK reporting rules require from March 2027.
Common mistakes
- Equating resilience with uptime SLAs → an SLA is a contractual credit, not a recovery capability; tolerances are about harm to customers.
- "AWS is designated a CTP/CTPP, so the regulators have that covered" → your obligations are unchanged.
- Exit plans that say "migrate to another cloud" → without data exports, key access, time estimates and rehearsal, it's not a plan.
- Multi-cloud for every service by default → huge cost and complexity, often lower resilience; decide per IBS against the tolerance.
- Only testing technology → scenarios must include people, processes and third parties (can staff sign in? who decides to fail over?).
- Ignoring detection and decision time → recovery plans look fine on paper and fail in incidents.
- Running chaos experiments without stop conditions → turn a test into an incident.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| FIS experiment fails: "no targets found" | Tasks lack the tag (tags not propagated) or selection mode too narrow | Enable tag propagation on the ECS service; check targets with aws ecs list-tasks and describe-tasks --include TAGS |
FIS AccessDenied stopping tasks | FIS role lacks permissions or the tag condition doesn't match | Check the role policy and resource tags |
| Experiment stops immediately | Stop-condition alarm already in ALARM state | Check the alarm before starting; tune TreatMissingData |
| Resilience Hub assessment says policy breached | Architecture can't meet the RTO/RPO you set (e.g. single-AZ database) | Apply recommendations or adjust targets with the service owner |
| Zonal shift not available for a load balancer | Resource type or configuration not supported, or cross-zone settings | Check ARC's supported resources and prerequisites ⚠ VERIFY |
| Scenario model shows everything within tolerance, but a real incident breached | Recovery times estimated, detection time ignored | Calibrate with measured test results and post-incident data |
Cost and cleanup
This lesson is free as written: regulatory concepts, local code, read-only CLI commands and IaC you don't have to deploy.
Optional costs: creating the FIS template, IAM role and alarm is effectively free; running an experiment is charged per action-minute (about USD 0.10 per action-minute ⚠ VERIFY) plus whatever the target service costs while running. Resilience Hub charges per application per month after any trial ⚠ VERIFY — don't add applications in the sandbox unless you'll delete them the same day.
# Remove the FIS template, role and alarm if you deployed them
cdk destroy --profile learn-sandbox # or: terraform destroy
aws fis list-experiment-templates --profile learn-sandbox
aws cloudwatch describe-alarms --alarm-names hl-m05-payments-5xx-during-experiment --profile learn-sandbox
# Remove any Resilience Hub application you created
aws resiliencehub delete-app --app-arn <app-arn> --profile learn-sandbox
# Confirm nothing tagged for the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m05-resilience --profile learn-sandboxExercises
- 🟢 Guided: Add a "Card authorisation" IBS (tolerance: 2 hours) to the resilience register with at least five components, including the card scheme connection and the tokenisation service from PCI DSS on AWS. Write tests for "AZ loss", "tokenisation vault database lost" and "card scheme link down".
- 🟡 Challenge: Extend
ScenarioEvaluatorto support sequential dependencies (the API can't recover until the ledger has), and a second-AZ loss scenario. Show how the result for "two AZs lost" changes. - 🔴 Live problem: The CIO forwards a board member's question: "Now that AWS is a designated critical third party, should we move half our services to a second cloud to reduce concentration risk?" Draft a one-page answer.
Assignment
Produce a resilience pack for the module project's Account API: the IBS it supports and an assumed tolerance (marked "for business approval"), a dependency map, the resilience register as code with tests for at least four scenarios, one FIS experiment template, and a two-page stressed-exit runbook. Commit it to your portfolio repository.
Solutions
Example components: Card authorisation API (Fargate, 3 AZs), tokenisation vault (DynamoDB, regional, survives AZ loss), KMS keys (regional), card scheme connection (scheme operator, recovery 1 h, workaround: stand-in processing by the scheme for low-risk transactions ⚠ business rule), fraud scoring (internal, workaround: rules-only scoring in 5 minutes).
Tests: AZ loss → zero disruption; vault database lost (service:dynamodb:eu-west-2 domain, recovery via point-in-time restore about 2–4 hours) → likely breaches a 2-hour tolerance unless restores are faster — a finding; scheme link down → within tolerance only with stand-in processing.
Why: card authorisation has a tighter tolerance than payments, so the same architecture can be fine for one IBS and a vulnerability for another.
Add IReadOnlyList<string> DependsOn to Component. Compute each affected component's finish time recursively: finish(c) = max(finish(d) for d in DependsOn affected) + time(c), memoised. The IBS disruption is detection time plus the maximum finish time. For "two AZs lost" with a three-AZ API and a two-AZ ledger, the ledger loses both its AZs, so it needs full recovery; the API (surviving in AZ c) has to wait for the ledger. The result moves from zero disruption to detection + ledger recovery, which shows why the ledger's AZ footprint matters.
Why: real recoveries are ordered; parallel models flatter you.
Key points: (1) CTP designation means regulators oversee AWS's resilience to the sector; it doesn't change Harbourline's own responsibilities, nor does it signal that using AWS is riskier. (2) Concentration risk should be managed per important business service against its tolerance, not by splitting services by count. (3) Our register shows single-AZ failures are absorbed; the main vulnerability is a Region-wide event for payments, which a warm standby in a second AWS Region (subject to DPO review) addresses at far lower cost and complexity than multi-cloud. (4) Multi-cloud adds identity, security, skills and operational complexity that can reduce resilience. (5) Where we do want provider independence is in the stressed-exit plan: data exports we control, portable workloads, a rehearsed restore outside AWS. Recommendation: invest in the Region-level vulnerability and the exit-plan rehearsal; revisit multi-cloud for specific IBS if tolerances or regulatory expectations change. The final decision sits with the board on the advice of risk.
Knowledge check
Answer: The maximum tolerable level of disruption to an important business service (usually a maximum duration, sometimes volume or customers affected), beyond which harm is intolerable. The firm sets it, with board approval; engineering designs and tests against it.
Answer: Nothing in terms of Harbourline's own obligations: it still owns its IBS mapping, tolerances, third-party risk management, contracts, testing, incident reporting and exit plans. Designation adds direct regulatory oversight of the provider's services to the sector.
Answer: A stressed exit is forced and sudden (provider failure, legal or sanctions issue, severe prolonged outage) and must keep important services running with little notice. A non-stressed exit is a planned, orderly migration. Plans for both should be documented and the stressed exit rehearsed.
Answer: AWS Fault Injection Service, with a narrowly targeted experiment template and a CloudWatch alarm as a stop condition. ARC zonal shift practice runs complement it by testing traffic movement away from an AZ.
Answer: The FCA (PS26/2) and PRA (PS7/26) published final rules on 18 March 2026; they apply from 18 March 2027.
Interview questions
- "How do you translate an impact tolerance into architecture?" — Start from the tolerance, subtract realistic detection and decision time, derive RTO/RPO per component, choose multi-AZ/multi-Region/backup patterns that meet them, then prove it with FIS experiments, failover drills and restore tests, and record vulnerabilities where you can't.
- "What should an exit plan for AWS contain?" — Triggers, stressed and non-stressed paths, how IBS continue meanwhile, data export and key access independent of the provider, portable workloads, substitutes for provider-specific services, time and cost estimates, roles, and rehearsal evidence.
- "Is multi-cloud the answer to concentration risk?" — Sometimes, for specific services, but usually multi-AZ plus multi-Region plus a credible exit plan achieves the tolerance with less complexity. Decide per IBS against tolerances and explain the reasoning.
- "What does DORA require that affects engineers?" — ICT risk management, incident classification and reporting, resilience testing (including threat-led penetration testing for some firms), a register of ICT third-party arrangements, specific contract terms, and exit strategies.
- "How do you know your resilience tests are realistic?" — Use severe but plausible scenarios, include people and third parties, measure actual recovery times, include detection time, and compare with real incidents.
Certification practice
A bank runs an ASP.NET Core API on Amazon ECS on AWS Fargate behind an Application Load Balancer across three Availability Zones. The resilience team must regularly prove the service continues serving traffic when tasks in one AZ fail, and the test must stop automatically if customer errors rise. What should the solutions architect recommend?
A. Manually terminate tasks with the ECS console once a quarter. B. Create an AWS FIS experiment template that stops tasks in one AZ, with a CloudWatch alarm on 5xx errors as a stop condition. C. Use AWS Trusted Advisor fault tolerance checks. D. Enable Amazon CloudWatch Synthetics canaries.
Answer: B. FIS runs controlled, repeatable experiments with automatic stop conditions. A isn't repeatable or safe. C checks configuration, not behaviour under failure. D monitors but doesn't inject faults.
A bank must be able to recover its ledger database even if an attacker obtains administrator access to the production account and tries to delete backups. Which approach meets this requirement?
A. Enable automated RDS snapshots in the production account. B. Copy AWS Backup recovery points to a vault in a separate account with AWS Backup Vault Lock, and test restores regularly. C. Enable Multi-AZ on the database. D. Store database credentials in AWS Secrets Manager with rotation.
Answer: B. Cross-account copies with Vault Lock protect backups from deletion by compromised production administrators; testing proves recoverability. A keeps backups in the compromised account. C protects against infrastructure failure, not malicious deletion. D is unrelated to backup protection.
An application behind an Application Load Balancer runs in three Availability Zones. During partial impairment of one AZ, the operations team wants to move traffic away from that AZ quickly without redeploying. Which feature should they use?
A. Amazon Route 53 weighted routing between AZs B. Amazon Application Recovery Controller zonal shift C. AWS Global Accelerator endpoint weights per Region D. ALB cross-zone load balancing turned off
Answer: B. Zonal shift temporarily moves traffic away from one AZ for supported load balancers. A — Route 53 records don't target AZs of a single ALB. C operates across Regions/endpoints, not AZs of one ALB. D doesn't move traffic away from an impaired AZ.
Completion checklist
- I can explain IBS, impact tolerances, mapping, scenario testing and self-assessment
- I can explain SS2/21-style outsourcing expectations and the difference between stressed and non-stressed exit
- I know what the CTP regime and DORA are, and that AWS EMEA SARL designation doesn't change the bank's obligations
- I ran the scenario evaluator tests and understood the Region-loss vulnerability
- I can describe an FIS experiment with a stop condition
- I removed anything I deployed and confirmed nothing tagged
Lab=m05-resilienceremains
Key takeaways
- UK regulators expect banks to stay within impact tolerances for important business services under severe but plausible disruption, and to prove it.
- Third-party risk management covers materiality, due diligence, contracts, continuity and exit; exit plans need data, keys, workloads, time estimates and rehearsal.
- AWS EMEA SARL is designated under both the UK CTP regime (from July 2026) and DORA (November 2025); the bank's responsibilities are unchanged.
- Manage concentration risk per service against tolerances; multi-AZ, multi-Region and a credible exit plan usually beat blanket multi-cloud.
- Test with FIS, ARC and restore drills, measure real recovery times, and feed vulnerabilities into the self-assessment.
Further reading
- FCA — Operational resilience
- PRA SS2/21 — Outsourcing and third party risk management
- FCA PS26/2 — Operational incident and third party reporting
- Regulation (EU) 2022/2554 (DORA) on EUR-Lex
- AWS designated as a critical third-party provider under the EU's DORA (AWS Security Blog)
- What is AWS Resilience Hub?
- What is AWS Fault Injection Service?
- AWS Well-Architected Financial Services Industry Lens
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Audit logging and evidence collection
Understand what auditors actually ask a bank for, and build evidence on AWS that answers them: AWS Artifact reports, CloudTrail and AWS Config as configuration and activity evidence, application audit events, and a repeatable .NET evidence collector that packages, hashes and locks what it gathers.
Module project: a secure banking API (field encryption, roles, audit, secrets and incident runbooks) with Labs 5.4 to 5.6
Run and extend the secure edition of the Harbourline Accounts API: field-level encryption bound to the customer, a blind index for IBAN search, role-based access with segregation of duties, audit events and masked logs, Terraform for KMS, secrets and a least-privilege role, plus the CloudTrail, GuardDuty and pipeline-scanning labs and two incident runbooks.