Disaster-recovery architecture for important business services
Map important business services and their impact tolerances to backup-and-restore, pilot-light, warm-standby or active-active designs, plan for cyber recovery and dependencies, and check the mapping with a runnable C# tolerance checker.
Assumes: .NET 10 LTS (file-based apps) · AWS SDK for .NET v4 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1 (DR example us-west-2). Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the chain from important business service to impact tolerance to RTO and RPO to a DR strategy.
- Choose between backup and restore, pilot light, warm standby and multi-site active-active for each service, and justify the cost.
- Find the hidden dependencies that break recovery: identity, DNS, keys, quotas, partners, on-premises links and pipelines.
- Distinguish DR for an Availability Zone or Region failure from cyber recovery, where replication copies the damage.
- Design DR tests that produce evidence of measured recovery times, not assumed ones.
- Run a C# checker that finds services whose chosen strategy cannot meet their tolerance, and mark regulatory points as needing review.
Prerequisites
- Availability vocabulary, Multi-AZ by default and Database failover
- AWS Backup in depth and Resilience Hub and game days
- Operational resilience and third-party risk
- Core banking integration for hybrid connectivity
- Tools: .NET 10 SDK or Docker for the tolerance checker
Why this matters in banking
When an entire cloud Region, a data centre or a critical dependency fails, a bank is judged on one question: how quickly did customers get the important services back, and how much data did they lose? Regulators in many jurisdictions expect firms to identify their important business services, set a maximum tolerable disruption for each, and show by testing that they can stay within it ⚠ VERIFY which regimes apply to your firm (for example UK operational-resilience rules and EU digital operational resilience rules).
Teams often confuse "we have Multi-AZ" with "we have disaster recovery". Multi-AZ survives the loss of a data centre in a Region. It does nothing for a regional event, a bad deployment that corrupts data, an expired certificate chain, a ransomware attack or an unavailable identity provider. A DR architecture answers each of those with a named strategy and evidence.
Educational content, not regulatory advice
Impact tolerances, severe-but-plausible scenarios, testing frequency and reporting duties are set by your regulators and your firm's risk appetite ⚠ VERIFY. The numbers in this lesson are illustrative planning values for the fictional Harbourline Bank, not recommendations, and not claims that a design meets any regulation.
The idea in plain language
Start from the customer, not the server:
- Important business service (IBS): something the bank provides whose failure could harm customers or the market: "make a retail payment", "view balance and statements", "access cash through cards".
- Impact tolerance: the maximum disruption you accept for that service, for example "no longer than 2 hours".
- RTO (recovery time objective): how long recovery is allowed to take, set inside the tolerance with margin.
- RPO (recovery point objective): how much recent data you can afford to lose.
- Strategy: the design that can actually meet the RTO and RPO.
- Evidence: test results that show it did.
Analogy: a hospital's emergency plan. "Emergency surgery must be possible within 30 minutes of a power cut" is the tolerance. A generator that starts in 10 seconds is the strategy. The monthly generator test, with the timing written in the log, is the evidence. A generator nobody has tested is a hope.
The four AWS strategies
| Strategy | What runs in the recovery Region | Typical recovery | Typical data loss | Relative cost |
|---|---|---|---|---|
| Backup and restore | Nothing; backups are copied there | Hours to a day or more | Since last backup | Lowest |
| Pilot light | Core data replicated; compute switched off or minimal | Tens of minutes to hours | Seconds to minutes | Low |
| Warm standby | Full stack at reduced capacity, scaled up on failover | Minutes | Seconds | Medium |
| Multi-site active-active | Full stack serving traffic in both Regions | Near zero for the Region loss | Near zero, with conflict handling | Highest, plus complexity |
The ranges are planning assumptions. Your measured times from tests replace them. Never put a typical value in front of a regulator as if it were your result.
Why the weakest dependency sets the time
A service is only as recoverable as its slowest or least-protected dependency. If payment initiation has a 2-hour tolerance but depends on a reporting database that restores in 8 hours, the dependency chain sets the real answer.
What problem the AWS services solve
| Need | Service or feature | Honest limits |
|---|---|---|
| Copy backups across Regions and accounts | AWS Backup cross-Region and cross-account copy | Restore speed depends on data size; test it |
| Backups safe from deletion and ransomware | AWS Backup Vault Lock, logically air-gapped vault | Vault Lock compliance mode is irreversible after the cooling-off period; plan carefully |
| Relational data across Regions | Aurora Global Database | Asynchronous replication, so RPO is not zero; typical lag is low but you must measure ⚠ VERIFY |
| NoSQL across Regions | DynamoDB global tables | Active-active writes need conflict thinking (last writer wins by default; check consistency options ⚠ VERIFY) |
| Object replication | S3 Cross-Region Replication, optionally with Replication Time Control | Replicates new objects after enabling; existing objects need batch replication; check delete-marker behaviour ⚠ VERIFY |
| Traffic failover | Route 53 health checks, Amazon Application Recovery Controller (ARC) routing controls | ARC Readiness Check is closed to new customers; use routing controls and your own tests ⚠ VERIFY status |
| Server-based workloads | AWS Elastic Disaster Recovery | Replicates servers; fits lift-and-shift estates more than cloud-native ones |
| Rebuild from code | CloudFormation, CDK, Terraform | Needs the pipeline and artefacts available in the DR Region |
| Multi-Region keys and secrets | KMS multi-Region keys, Secrets Manager replication | Replication must be set up before the disaster |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Site Recovery | AWS Elastic Disaster Recovery | Both replicate servers to a recovery location |
| Azure Backup, Recovery Services vault | AWS Backup with vaults and Vault Lock | Vault Lock gives WORM protection |
| Azure paired regions | Chosen Region pair (no automatic pairing) | You pick the DR Region, including data-residency checks |
| Azure Traffic Manager / Front Door failover | Route 53 failover, ARC routing controls, CloudFront origin failover | ARC is the high-availability control-plane pattern |
| SQL geo-replication, failover groups | Aurora Global Database, RDS cross-Region replicas | Aurora Global promotes a secondary Region |
| Cosmos DB multi-region writes | DynamoDB global tables | Similar active-active model with conflict resolution |
| Azure Chaos Studio | AWS Fault Injection Service | Experiments with stop conditions and IAM scoping |
Architecture
Security-boundary view
The recovery account's backups cannot be deleted by workload administrators, which protects against ransomware and insider deletion. Failover itself is a controlled, authenticated, logged action.
The 15-section design
The fictional case: Harbourline's important business services and a strategy for each.
Sections 1 to 3: requirements
| Section | Content |
|---|---|
| 1 Business | Keep retail payments, balance and statement access, and card use within agreed impact tolerances during severe but plausible disruptions, and be able to evidence this |
| 2 Functional | F1 Fail over payment initiation to a second Region. F2 Restore from immutable backups after a cyber event. F3 Fail back after recovery. F4 Communicate status to customers. F5 Run regular tests with timed results |
| 3 Non-functional | See the service mapping below. Test cadence: failover test twice a year, restore test quarterly ⚠ VERIFY with risk. Evidence retained 7 years ⚠ VERIFY |
Illustrative service mapping:
| Important business service | Impact tolerance | RTO target | RPO target | Strategy | Why |
|---|---|---|---|---|---|
| Retail payment initiation | 2 hours | 30 minutes | 1 minute | Warm standby | Tolerance short; asynchronous replication is acceptable if reconciliation covers the gap |
| Balance and statement view | 1 hour | 30 minutes | 5 minutes | Warm standby | Read-heavy; replica data serves reads quickly |
| Card authorisation | 15 minutes | 5 minutes | 1 minute | Active-active (⚠ business decision) | Shortest tolerance; may rely on scheme stand-in processing ⚠ VERIFY |
| Regulatory reporting | 3 days | 24 hours | 24 hours | Backup and restore | Batch, tolerant; restore tested |
| Customer notifications | 8 hours | 4 hours | 1 hour | Pilot light | Queue backlog can be replayed |
Section 5: service selection
| Choice | Why | Rejected alternative | Trade-off |
|---|---|---|---|
| Aurora Global Database for the ledger | Cross-Region replication with fast promotion | Cross-Region snapshots only | Higher cost; non-zero RPO to be covered by reconciliation |
| DynamoDB global tables for idempotency and state | Multi-Region replicas with low lag | Single-Region table with backups | Conflict behaviour must be understood |
| Warm standby for payments | Meets 30-minute RTO at moderate cost | Active-active | Failover is a manual or semi-automatic decision |
| AWS Backup with a locked recovery-account vault | Protection against deletion and ransomware | Backups in the workload account | Another account and key to manage |
| Route 53 ARC routing controls for failover | Reliable data-plane control with approvals | Manual DNS edits | Setup and rehearsal |
Sections 6 and 7: network and security
- Network. Mirror the VPC layout in the DR Region with non-overlapping CIDRs if you plan cross-Region connectivity. On-premises links must exist in the DR Region too: a second Direct Connect location or VPN to the DR Region, or the core is unreachable after failover (core banking integration).
- Identity. Prefer workforce identity that survives a Region event. Keep a break-glass role, tested, stored under dual control. Remember that IAM control-plane operations should not be required during recovery: pre-create roles, policies and keys.
- Keys. KMS multi-Region keys or separately replicated key material for encrypted replicas. Test decryption in the DR Region.
- Secrets and certificates. Replicate secrets; ensure certificates for DR hostnames are issued and not about to expire.
- Data residency. The DR Region must also satisfy data-residency and transfer rules ⚠ VERIFY with your data-protection officer. A UK bank might pair two Regions that meet its own policy; this lesson uses us-east-1 and us-west-2 for the lab only.
- Cyber recovery. Replication faithfully copies corruption and malicious deletes. Keep immutable, versioned, delayed-copy backups in a separate account, and rehearse restoring into an isolated clean environment.
Sections 8 to 10: API, data and flow
- Failover API (internal, privileged).
POST /dr/failoverrequires two approvals and a reason; it flips the routing control, scales the standby, promotes the database and records a timeline. Never expose it to customers. - Database. A
dr_eventstable records every step with timestamps. The timeline is the evidence: detection, decision, each action and the time customers regained service. - Failover flow. (1) Detect via synthetic checks and alarms. (2) Incident commander decides against pre-agreed criteria. (3) Freeze deployments. (4) Promote database. (5) Scale standby. (6) Flip routing. (7) Verify with synthetic payments. (8) Communicate. (9) Later: reconcile any replication gap, then fail back deliberately.
Section 11: failure scenarios (of the DR design itself)
| Failure | Detection | Automatic response | Impact | Manual step |
|---|---|---|---|---|
| Replication lag larger than RPO | Lag alarm | Alert | Data-loss risk at failover | Decide with business; reconcile |
| DR Region quotas too low | Pre-failover capacity test | None | Scale-up fails | Raise quotas in advance |
| Secrets or keys not replicated | DR test | None | Standby cannot start | Fix replication, retest |
| DNS or routing change not effective | Synthetic probes | None | Customers still hit failed Region | Lower TTLs; verify |
| Partner allow-lists exclude DR addresses | DR test | None | Partner calls fail | Pre-register DR egress IPs |
| Split brain (both Regions write) | Monitoring | Fencing | Data conflicts | Stop one side; reconcile |
| Ransomware corrupts data and replicas | Anomaly or integrity checks | Backups immutable | Primary and replica unusable | Restore clean point in recovery account |
| Runbook out of date | Game-day failures | None | Slow recovery | Update after each test |
| Failback causes second outage | Test | None | Extra downtime | Rehearse failback; plan window |
| Control-plane dependency during recovery | Test with simulated outage | Static stability design | Delay | Pre-provision; avoid creating resources during the event |
Sections 12 to 15
- Scalability. In warm standby the standby is smaller, so the recovery plan includes scaling up; verify quotas and capacity in the DR Region, and consider reserved capacity for critical services ⚠ VERIFY options.
- Cost. Estimate each strategy's recurring cost: replicated storage, cross-Region data transfer, standby compute, backup storage and copies, and test run cost. Relative cost grows with strategy tier; active-active also pays for operating two live Regions. Compare with the business cost of an hour of downtime for each service ⚠ VERIFY prices.
- Implementation. Phase 1 service inventory, tolerances and dependency maps. Phase 2 backups with vault lock and cross-account copy. Phase 3 data replication. Phase 4 standby stack from IaC. Phase 5 routing controls and approvals. Phase 6 timed tests and runbooks. Phase 7 cyber-recovery rehearsal.
- Compliance touchpoints.
| Area | Where it touches | Reviewing team | Evidence |
|---|---|---|---|
| Operational resilience: important business services and impact tolerances ⚠ VERIFY | Mapping table, tolerances | Operational resilience, risk | Mapping, approved tolerances, scenario results |
| Testing obligations | Test cadence and scope | Risk, internal audit | Timed test reports |
| Third-party and cloud concentration risk | Cloud provider and partner dependencies | Third-party risk | Exit and stress-exit plans |
| Data residency and transfer | DR Region choice | Data protection officer, legal | Region decision record |
| Backup retention and immutability | Vault lock retention | Compliance | Retention schedule |
Step-by-step implementation
Inventory important business services and dependencies
For each service list every component, data store, partner and network link it needs. Draw the dependency graph and mark the slowest-to-recover item.
Agree tolerances with the business, then derive RTO and RPO
Tolerance comes from the business and risk function. Set RTO inside the tolerance with margin for detection and decision time.
Pick the cheapest strategy that meets the numbers
Use the checker in this lesson to compare each service's choice against its needs, then record the choice and the rejected alternatives in an ADR.
Build the protection layers
Backups with cross-Region and cross-account copy and a locked vault. Replication for data stores. KMS and secret replication. IaC for the standby.
Script failover and failback
Write a runbook with owners, commands and checkpoints. Automate what you can, but keep the decision with a named person.
Test, time, record, improve
Run a game day with timers. Record detection time, decision time, action time and the time to a verified payment. Fix every gap and update the runbook.
AWS Console instructions
- Console → AWS Backup → Backup vaults → Create backup vault. Pick a KMS key, then Vault lock to set minimum retention (and, in a real recovery account, a cooling-off period for compliance mode).
- Console → AWS Backup → Backup plans → Create plan. Add a rule with a schedule, a retention, and Copy to destination with a Region and vault.
- Console → RDS → Databases → your Aurora cluster → Actions → Add AWS Region to create a global database secondary (billing starts).
- Console → DynamoDB → Tables → Global tables → Create replica for a global table replica.
- Console → Route 53 Application Recovery Controller to review routing controls where available ⚠ VERIFY current console layout.
AWS CLI
# Create a vault in the DR Region and a plan copy rule is then added in the plan (see IaC).
aws backup create-backup-vault --backup-vault-name hl-m14-dr-vault \
--backup-vault-tags Project=cea-lab,Lab=m14-l16,Environment=learn,Owner=learner \
--profile learn-sandbox --region us-west-2
# Create a logically air-gapped vault (retention bounds are required).
aws backup create-logically-air-gapped-backup-vault --backup-vault-name hl-m14-airgap \
--min-retention-days 7 --max-retention-days 35 \
--profile learn-sandbox --region us-west-2
# See recent copy jobs: did the cross-Region copy actually happen?
aws backup list-copy-jobs --by-state COMPLETED --max-results 5 \
--profile learn-sandbox --region us-east-1
# Check Aurora global cluster membership (read-only).
aws rds describe-global-clusters --profile learn-sandbox --region us-east-1
# Add a DynamoDB replica in the DR Region (billing starts for the replica).
aws dynamodb update-table --table-name hl-m14-idempotency \
--replica-updates '[{"Create":{"RegionName":"us-west-2"}}]' \
--profile learn-sandbox --region us-east-1Cross-Region replication and global databases bill for as long as they exist; treat them as design-only unless you will delete them within the hour ⚠ VERIFY prices. Check each command with help in your CLI version.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.Backup;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.KMS;
using Constructs;
namespace Harbourline.Architecture.DisasterRecovery;
public class DrBackupStack : Stack
{
public DrBackupStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var key = new Key(this, "BackupKey", new KeyProps { EnableKeyRotation = true });
var vault = new BackupVault(this, "PrimaryVault", new BackupVaultProps
{
BackupVaultName = "hl-m14-primary-vault",
EncryptionKey = key,
// Min retention without a cooling-off period gives Governance-style lock; production
// compliance mode (with a cooling-off period) is a risk and compliance decision.
LockConfiguration = new LockConfiguration { MinRetention = Duration.Days(7) },
RemovalPolicy = RemovalPolicy.DESTROY,
});
// The destination vault lives in the DR Region (created there, see the CLI step).
var drVault = BackupVault.FromBackupVaultArn(this, "DrVault",
$"arn:aws:backup:us-west-2:{Aws.ACCOUNT_ID}:backup-vault:hl-m14-dr-vault");
var plan = new BackupPlan(this, "DailyPlan", new BackupPlanProps
{
BackupPlanName = "hl-m14-daily",
BackupVault = vault,
});
plan.AddRule(new BackupPlanRule(new BackupPlanRuleProps
{
RuleName = "daily-with-dr-copy",
ScheduleExpression = Schedule.Cron(new CronOptions { Hour = "3", Minute = "0" }),
DeleteAfter = Duration.Days(35),
CopyActions =
[
new BackupPlanCopyActionProps
{
DestinationBackupVault = drVault,
DeleteAfter = Duration.Days(35),
},
],
}));
// Anything tagged Backup=daily is protected.
plan.AddSelection("TaggedResources", new BackupSelectionOptions
{
Resources = [BackupResource.FromTag("Backup", "daily")],
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m14-l16");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "learner");
}
}A vault with a lock configuration may be hard to delete during cleanup; use a very short minimum retention in the sandbox and expect deletion to wait for retention.
Complete code
The checker below compares each service's impact tolerance and data-loss tolerance with what its chosen strategy can deliver, and proposes the cheapest strategy that would meet both. The strategy numbers are planning assumptions: replace them with measured values from your tests.
// Run with: dotnet run DrTolerance.cs
// Compares each important business service's impact tolerance and data-loss tolerance with what its
// chosen DR strategy can deliver. The strategy numbers are PLANNING ASSUMPTIONS: replace them with
// the times you measure in game days.
var strategies = new Dictionary<string, Strategy>
{
["backup-restore"] = new("backup-restore", RtoMinutes: 24 * 60, RpoMinutes: 24 * 60, RelativeCost: 1),
["pilot-light"] = new("pilot-light", RtoMinutes: 4 * 60, RpoMinutes: 15, RelativeCost: 2),
["warm-standby"] = new("warm-standby", RtoMinutes: 30, RpoMinutes: 1, RelativeCost: 4),
["active-active"] = new("active-active", RtoMinutes: 2, RpoMinutes: 1, RelativeCost: 8),
};
var services = new[]
{
new Service("Payment initiation", ToleranceMinutes: 120, DataLossMinutes: 1, Chosen: "pilot-light"),
new Service("Account balances", ToleranceMinutes: 60, DataLossMinutes: 5, Chosen: "warm-standby"),
new Service("Regulatory reporting", ToleranceMinutes: 3 * 24 * 60, DataLossMinutes: 24 * 60, Chosen: "backup-restore"),
new Service("Notifications", ToleranceMinutes: 8 * 60, DataLossMinutes: 60, Chosen: "backup-restore"),
};
var gaps = 0;
foreach (var s in services)
{
var chosen = strategies[s.Chosen];
var meets = chosen.RtoMinutes <= s.ToleranceMinutes && chosen.RpoMinutes <= s.DataLossMinutes;
var cheapest = strategies.Values.Where(x => x.RtoMinutes <= s.ToleranceMinutes && x.RpoMinutes <= s.DataLossMinutes)
.OrderBy(x => x.RelativeCost).FirstOrDefault();
if (!meets) gaps++;
Console.WriteLine($"{s.Name,-22} chosen={s.Chosen,-15} {(meets ? "MEETS" : "GAP ")} cheapest that meets: {cheapest?.Name ?? "none (redesign)"}");
}
// Downtime budget arithmetic for an availability target (30-day month).
foreach (var target in new[] { 99.9, 99.95, 99.99 })
Console.WriteLine($"{target}% availability = {(100 - target) / 100 * 30 * 24 * 60:0.0} minutes of downtime per 30 days");
return gaps == 0 ? 0 : 1;
public sealed record Strategy(string Name, int RtoMinutes, int RpoMinutes, int RelativeCost);
public sealed record Service(string Name, int ToleranceMinutes, int DataLossMinutes, string Chosen);Expected output (the program exits with code 1 because it finds gaps, which is useful in a CI step):
Payment initiation chosen=pilot-light GAP cheapest that meets: warm-standby
Account balances chosen=warm-standby MEETS cheapest that meets: warm-standby
Regulatory reporting chosen=backup-restore MEETS cheapest that meets: backup-restore
Notifications chosen=backup-restore GAP cheapest that meets: pilot-light
99.9% availability = 43.2 minutes of downtime per 30 days
99.95% availability = 21.6 minutes of downtime per 30 days
99.99% availability = 4.3 minutes of downtime per 30 daysCode explained line by line
Strategyholds the planning RTO and RPO (in minutes) and a relative cost index, not a price. The index lets the program rank options without pretending to know your bill.Servicecaptures the three facts the business gave you: impact tolerance, data-loss tolerance and the strategy currently chosen.- The dictionary initialiser uses named arguments (
RtoMinutes: 24 * 60) so each number's meaning is visible at the call site. meetsis true only when both the RTO is within the tolerance and the RPO is within the data-loss tolerance. Passing one is not enough.cheapestfilters strategies that meet both conditions and sorts byRelativeCost.FirstOrDefaultreturnsnullif none qualifies, which the output reports as "redesign": a requirement that no listed strategy can satisfy needs a different approach.- The payments row shows a gap: pilot light's planning RTO (4 hours) exceeds the 2-hour tolerance. Notifications show a gap because 24 hours of backup-and-restore exceeds their 8-hour tolerance.
- The availability loop computes the downtime budget:
(100 - target) / 100 * 30 * 24 * 60minutes in a 30-day month. These are the numbers to put next to an availability claim. return gaps == 0 ? 0 : 1makes the checker usable as a pipeline gate: a design that fails its own tolerance cannot merge.
Security considerations
⚠ Learning shortcut — not for production
The lab vault uses a short minimum retention, a single account and a tag-based selection. Production recovery uses a separate recovery account with its own administrators, a compliance-mode vault lock after a cooling-off period agreed with risk, multi-Region encryption keys, tested restore permissions that do not rely on the workload account, and a break-glass role stored under dual control.
- Protect the failover lever. Failover can itself cause an outage or data loss. Require two approvals, log everything, and allow only a small named group.
- Assume the attacker is in the primary Region. Replicated data may be corrupted. Immutable backups in another account are the line of last defence.
- Test restoring into isolation. A clean-room account lets you scan restored data before reconnecting it.
- Do not widen security to make DR easier. Pre-provision narrow roles in the DR Region rather than granting broad permissions "for the emergency".
- Keep DR runbooks safe but reachable. They must be accessible when your primary Region and identity provider are down, yet not public.
Banking use case
During a game day, Harbourline simulates the loss of the primary Region at 10:00. Synthetic probes alarm at 10:03. The incident commander, with the head of payments as second approver, declares failover at 10:08 according to pre-agreed criteria. The runbook promotes the Aurora secondary, scales the standby from 4 to 40 tasks, flips the routing control and runs 20 synthetic payments. Customers see payments succeed again at 10:41, 33 minutes after the decision and 41 minutes after the start. Against a 30-minute target RTO the test shows a gap of 11 minutes, mostly caused by a DR-Region vCPU quota that limited scale-up. The team raises the quota, adds a pre-warmed capacity step and retests in the next quarter. The timeline and the gap analysis are the evidence for the operational-resilience file ⚠ VERIFY what your firm must submit.
Common mistakes
- Equating Multi-AZ with DR. It protects against a data-centre failure only.
- Setting RTO equal to tolerance. Detection and decision time consume part of it; leave margin.
- Forgetting quotas in the DR Region. Quotas are per Region and are not replicated.
- Untested failback. Recovering in the DR Region is half the job; returning without data loss is the other half.
- Backups in the same account as the workloads. One compromised admin deletes both.
- Ignoring partners and on-premises links. The core or a screening provider may not accept calls from the DR Region.
- Assuming typical RTO figures apply to you. Only measured results count.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Cross-Region copy job fails | Destination vault missing, KMS key not usable in the destination, or role permissions | Create the vault first; check key policy and the plan role |
| Restore from backup is far slower than planned | Data volume larger than the test, or throughput limits | Re-time with production-scale data; consider continuous backup or replicas |
| Aurora secondary promotion succeeds but app cannot connect | Secrets, endpoints or security groups not mirrored | Replicate secrets, use a Region-agnostic connection abstraction, mirror networking |
| DNS failover does not move traffic | High TTL or health check misconfiguration | Lower TTL ahead of time; test health checks; use routing controls |
| Standby cannot scale up | Service quota or capacity limit in the DR Region | Raise quotas in advance; reserve capacity where necessary |
| Decryption fails in the DR Region | Single-Region key | Use multi-Region keys and test decryption |
| Vault cannot be deleted | Vault Lock retention active | Wait out retention in the lab; use short retention only |
Cost and cleanup
The C# checker is free. Everything else here is design-only by default: Aurora Global Database secondaries, DynamoDB replicas, cross-Region backup copies and warm-standby compute bill continuously ⚠ VERIFY. If you deploy the backup stack, create small vaults, use a tiny test resource, and delete them the same day. Vault Lock retention can delay deletion.
# Delete recovery points before the vault (list first to get ARNs).
aws backup list-recovery-points-by-backup-vault --backup-vault-name hl-m14-dr-vault \
--profile learn-sandbox --region us-west-2
aws backup delete-backup-vault --backup-vault-name hl-m14-dr-vault \
--profile learn-sandbox --region us-west-2
# Remove replicas and global clusters if you created any, then tear down IaC.
aws dynamodb update-table --table-name hl-m14-idempotency \
--replica-updates '[{"Delete":{"RegionName":"us-west-2"}}]' \
--profile learn-sandbox --region us-east-1
cdk destroy --profile learn-sandbox
terraform destroy
# Confirm nothing tagged for the course remains, in both Regions.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-west-2Exercises
- 🟢 Guided: run the checker, then change Payment initiation to
warm-standbyand Notifications topilot-light. Predict the exit code and the output. - 🟡 Challenge: extend the checker to read measured RTO and RPO from a JSON test-results file and flag any service whose measured time exceeds its target, even if the planning value passed.
- 🔴 Live problem: a ransomware scenario encrypts the primary database and the replicated copy within minutes. Write the recovery plan: what you restore from, which account performs it, how you decide the clean restore point, how you test the restored data before reconnecting, and what you tell the board about the RPO.
Assignment
Add architecture/disaster-recovery/ to your portfolio: a service-to-tolerance mapping table for five fictional services, the 15-section document for one of them, a dependency map highlighting the slowest-to-recover component, an ADR choosing warm standby over active-active for payments, a runbook with timing checkpoints, and the checker with unit tests.
Solutions
With payments on warm-standby (RTO 30, RPO 1) it meets 120 minutes and 1 minute, and notifications on pilot-light (RTO 240, RPO 15) meet 480 minutes and 60 minutes. All four rows are MEETS, the availability lines are unchanged and the exit code is 0. Why: the exit code depends only on the count of gaps, so removing both gaps makes it succeed.
Add a measured dictionary loaded from JSON ({ "Payment initiation": { "rtoMinutes": 41, "rpoMinutes": 1 } }). For each service, compare measured.RtoMinutes with the service's RTO target (not the tolerance) and flag TEST-GAP when it is larger. Prefer measured values over planning values whenever both exist. Why: regulators and your board care about what you demonstrated, so planning numbers should only be a starting hypothesis.
Restore from the locked backup vault in the recovery account, choosing a recovery point from before the first sign of compromise (use detection timelines from GuardDuty, CloudTrail and application anomalies to bracket it). Restore into an isolated clean-room account, scan and validate (checksums, row counts, reconciliation against partner statements), then promote through a controlled cut-over. Tell the board the actual RPO is the age of the chosen recovery point plus any transactions that must be re-created from partner and scheme records, and that reconciliation covers the gap. Replication does not help here because it copied the damage.
Knowledge check
Answer: Impact tolerance is the maximum disruption the business accepts for a service. RTO is the engineering target for recovery, set inside the tolerance to leave margin for detection and decision time.
Answer: It protects against the loss of a data centre within one Region. It does not cover regional events, data corruption, bad deployments or cyber attacks.
Answer: Backup and restore. Recovery takes hours or longer and data loss equals time since the last backup.
Answer: Replication copies encrypted or deleted data to the replica. Immutable backups that workload administrators cannot alter give a clean recovery point.
Answer: Quotas are per Region. A standby that cannot scale up to full capacity will not meet its RTO.
Interview questions
- How do you decide between warm standby and active-active? Compare the tolerance, RTO and RPO with each strategy's measured performance and cost; choose the cheapest that meets them and accept the complexity only when the tolerance demands it.
- What are the hidden dependencies in a Region failover? Identity, DNS, keys and secrets, certificates, quotas, partner allow-lists, on-premises connectivity, deployment pipelines and runbook access.
- How is cyber recovery different from DR? The damage may be replicated, so you need immutable, delayed, separate-account backups and clean-room restoration.
- How do you prove your DR works? Timed tests with recorded timelines, gap analysis, fixes and retests, retained as evidence.
- Who decides to fail over? A named incident commander against pre-agreed criteria, with a second approver, because failover has its own risk.
Certification practice
A payment service needs an RTO of 30 minutes and an RPO of under a minute across a Region failure, at reasonable cost. Which strategy is the best fit?
A. Backup and restore with daily snapshots. B. Pilot light with compute switched off and manual rebuild. C. Warm standby with replicated data stores and a reduced-capacity stack ready to scale up. D. Single-Region Multi-AZ deployment.
Answer: C. It combines continuous replication with a running stack. A and B cannot meet the RTO or RPO, and D does not survive a Region failure.
Which control best protects backups from deletion by a compromised workload administrator?
A. Backups in the same account with a tag. B. Cross-account copies to a vault in a recovery account with Backup Vault Lock. C. Longer snapshot names. D. A weekly email report.
Answer: B. Separate administration plus a lock prevents deletion. A, C and D do not restrict deletion.
A team plans failover to a second Region but never tested scaling in that Region. What is the most likely cause of a missed RTO?
A. Too many Availability Zones. B. Service quotas in the DR Region that are too low for the failover capacity. C. Using Parquet files. D. Enabling encryption.
Answer: B. Quotas are per Region, and unplanned limits are a classic failover surprise. The others do not typically prevent scale-up.
Completion checklist
- I can chain important business service, impact tolerance, RTO, RPO and strategy
- I ran the checker and understood why two services show gaps
- My dependency map names the slowest-to-recover component
- My plan separates AZ or Region DR from cyber recovery
- I ran cleanup in both Regions and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Start from the customer service and its impact tolerance, then derive RTO, RPO and strategy.
- Choose the cheapest strategy that meets the numbers, and replace planning figures with measured test results.
- Dependencies (identity, keys, quotas, partners, networks) usually decide real recovery time.
- Cyber recovery needs immutable backups in a separate account and a clean-room restore.
- Tolerances, testing duties and Region choices need risk, compliance and data-protection review.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Audit and compliance architecture
Design a tamper-evident audit trail with CloudTrail, a locked log archive, application audit records and evidence collection, answer who-accessed-what questions with Athena, and verify a hash-chained audit log in C#.
Module 14 cheat sheet
One-page revision summary for Advanced Banking Architecture: services, Azure mapping, CLI commands, limits, decision rules, exam traps and cost traps.