The four DR strategies: backup and restore, pilot light, warm standby and active-active
Compare the four disaster-recovery strategies against RTO, RPO and cost, choose one per business service, and understand what each demands from your infrastructure code, data replication, quotas and runbooks.
Assumes: .NET 10 LTS · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · primary us-east-1, DR us-west-2 (design-only). Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Describe backup and restore, pilot light, warm standby and multi-site active-active (including hot standby) and the difference between each pair of neighbours.
- Pick a strategy for a banking service from its RTO, RPO, regulatory expectations and budget, and justify it in a paragraph.
- Name the AWS services that implement each strategy's data, compute and traffic layers.
- Explain why infrastructure as code, a separate account, quotas in the DR Region and data-plane-only failover decide whether a strategy works on the day.
- Distinguish a Regional disaster from a data disaster, and say which strategies help with each.
- Use the lab's strategy selector and runbook timer to check a design on paper.
Prerequisites
- Availability vocabulary — RTO, RPO, impact tolerance, control plane versus data plane
- Multi-AZ by default — what HA already gives you inside one Region
- Operational resilience and third-party risk — why regulators ask for tested scenarios
- UK GDPR and data residency — where a DR copy of personal data may live
- Tools: Docker for the lab tests; no AWS account needed (this lesson is a design lesson)
Why this matters in banking
A regional event takes a core AWS Region out of service for several hours. Harbourline has four important business services affected, each with a different story:
- Card authorisation must keep approving within seconds. It runs in two Regions at once, and customers barely notice.
- Online banking goes down for 20 minutes while a scaled-down copy in the second Region is scaled up and traffic is switched.
- Statements are unavailable for most of a day, which is acceptable because the data was backed up and a rebuild from infrastructure code takes hours.
- The reporting warehouse is rebuilt from backups over the weekend and nobody outside the bank notices.
That is not luck. Each service was given a strategy that matches its tolerance, and each strategy was tested. The alternative, one expensive answer for everything or one cheap answer for everything, either wastes a large part of the infrastructure budget or breaches a regulatory impact tolerance on a bad day.
The idea in plain language
AWS describes four disaster-recovery strategies on a spectrum, from cheap and slow to expensive and fast. In every one, the primary Region is where traffic normally goes and a recovery Region (also called the DR Region) holds what you need to recover. What differs is how much of the workload is already running there.
Analogy: the bank's head office floods.
- Backup and restore: nightly copies of records are in a safe in another city. To reopen you must find a building, fit it out, and load the records: hours to days.
- Pilot light: a rented office with the filing cabinets installed and phones connected, but the lights off and no staff. Staff arrive and switch it on: tens of minutes to an hour or so.
- Warm standby: a smaller working branch with a skeleton crew, all systems on. You add staff and redirect customers: minutes.
- Active-active: two full branches serve customers every day. If one closes, the other already has the customers: seconds to near zero.
The four strategies in detail
| Backup and restore | Pilot light | Warm standby | Multi-site active-active | |
|---|---|---|---|---|
| What runs in the DR Region | Nothing, only backups stored there | Data stores replicated and running; compute off (not deployed or desired count 0) | A scaled-down, fully working copy | A full-capacity copy serving live traffic |
| Data | Periodic backups copied across Regions | Continuous replication to live stores | Continuous replication | Continuous replication in both directions or partitioned writes |
| Typical RPO | Backup interval plus copy time (hours) | Replication lag (seconds to minutes) | Replication lag (seconds to minutes) | Near zero (still not zero for asynchronous data ⚠ VERIFY) |
| Typical RTO | Hours or more | Tens of minutes to hours | Minutes | Near zero, no failover step |
| Recovery action | Restore data, deploy everything from IaC, switch traffic | Start or scale compute, switch traffic | Scale up, switch traffic | Remove the failed Region from traffic |
| Relative cost | Lowest (storage and transfer) | Low to moderate (live data stores) | Moderate to high | Highest, plus complexity |
| Hardest part | Restore speed and IaC completeness | Keeping the "off" part deployable | Keeping the standby really working | Data consistency and write conflicts |
The RPO and RTO words in the table are the usual orders of magnitude from AWS's disaster-recovery guidance, not guarantees: your numbers come from your architecture and your drills ⚠ VERIFY the wording in the current whitepaper before quoting them in a design document.
Pilot light versus warm standby. The AWS whitepaper puts it neatly: both have a copy of the environment in the recovery Region, but pilot light cannot process requests without extra steps (turn servers on, possibly deploy non-core parts, scale up), whereas warm standby can handle traffic immediately at reduced capacity and only needs scaling up.
Hot standby versus active-active. Both run a full-capacity environment in the second Region. In hot standby it receives no user traffic until failover; in active-active it serves users all the time, so there is no "failover" at all, only the loss of a Region.
Regional disaster versus data disaster
Two different disasters need different defences:
| Disaster | Example | Helps | Does not help |
|---|---|---|---|
| Regional | A Region-wide service impairment | All four strategies (that is what they are for) | Backups in the same Region |
| Data | A bad migration, deletion, ransomware, a bug that corrupts rows | Point-in-time recovery and backups, immutable and isolated copies | Replication (it copies the damage) and active-active (both Regions get it) |
AWS's own guidance makes this point: even active-active needs backups, because a data disaster has a recovery time greater than zero and a recovery point before the damage was discovered. A DR strategy that does not include backups with a tested restore is incomplete, whichever of the four it is.
What decides whether a strategy works on the day
- Infrastructure as code for the recovery Region. If the DR environment cannot be created from code, backup and restore becomes weeks of work and pilot light is a gamble. Deploy the same stacks to both Regions with a parameter for "scaled down" or "off".
- Separate accounts (or at least separate blast radius). AWS recommends a different account per Region for the highest isolation, in case compromised credentials are part of the disaster. At minimum, the DR copy must not share the credentials that an attacker could use to delete the primary.
- Quotas in the recovery Region. Service quotas (vCPU, Fargate tasks, Elastic IPs, Lambda concurrency) are per Region. A pilot light that cannot scale to production size because the quota is 10 percent of what you need is not a pilot light.
- Data-plane failover. Prepare using the control plane (create the environment, set up replication), execute using the data plane (health-check-driven DNS, routing controls, Global Accelerator). Scaling out an Auto Scaling group is itself a control-plane action, so a design that must scale from near zero during a disaster depends on it. Sufficient pre-provisioned capacity (statically stable "hot standby") removes that dependency; AWS describes it as a trade-off.
- Detection and decision. Failing over costs time and risk, and a false alarm still costs you. AWS suggests automatic failover based on health checks or alarms be used with caution, with a manually initiated but fully scripted failover often preferred: the "push of a button" model.
- Failback. After the event you must return, often by re-establishing replication in the opposite direction. Failback is a second migration, and needs its own runbook and rehearsal.
- People and communication. Who declares, who executes, who talks to customers and the regulator. Impact tolerance is breached by silence as often as by technology.
What problem the AWS service solves
No single AWS service is "DR". The strategies are assembled from services you have met, and the table shows the usual pieces:
| Layer | Backup and restore | Pilot light | Warm standby | Active-active |
|---|---|---|---|---|
| Backups | AWS Backup with cross-Region and cross-account copies; EBS snapshots; RDS and DynamoDB backups | Same, plus point-in-time backups of replicated data | Same | Same |
| Relational data | Snapshot copy and restore | Aurora Global Database or RDS cross-Region read replica | Same | Aurora Global Database (one writer, write forwarding) or Aurora DSQL ⚠ VERIFY suitability, or partitioned writes |
| NoSQL data | DynamoDB backup restore | DynamoDB global tables | Same | DynamoDB global tables (multi-active) |
| Objects | S3 versioning, cross-Region replication | S3 replication | S3 replication | S3 replication |
| Compute | Redeploy from IaC; AMIs copied across Regions | Deployed but at zero or switched off; AWS Elastic Disaster Recovery for EC2 and on-premises servers | Running at reduced capacity | Running at full capacity |
| Traffic | Route 53 record change | Route 53 failover; ARC routing control | Same | Route 53 latency or weighted routing, Global Accelerator |
| Testing | Scheduled restore tests | Failover drills | Failover drills, continuous smoke tests | Region-loss game days |
Limitations to be honest about:
- Replication is asynchronous for most cross-Region data services, so RPO is the replication lag, not zero. Monitor the lag as a first-class metric.
- Not every service replicates the way you want. Some data stores, secrets and certificates need explicit cross-Region setup (for example, multi-Region keys and secret replication). Inventory dependencies, including identity and DNS.
- A DR Region is a second production environment for security, patching and cost purposes.
- Residency rules apply to the second Region for personal data; Harbourline's UK and EU customers constrain the choice.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Site Recovery (ASR) to a paired Region | AWS Elastic Disaster Recovery (EC2 and on-premises) | Elastic Disaster Recovery is pilot-light-style: continuous block-level replication into a low-cost staging area, launching full capacity on failover |
| Azure Backup (Recovery Services vault, GRS) | AWS Backup vaults with cross-Region and cross-account copies | Vault Lock gives WORM protection; restore testing is a feature (AWS Backup in depth, lesson in this module) |
| Paired Regions with automatic geo-redundant storage | You choose the DR Region; S3 replication is configured per bucket | No automatic pairing: that is both freedom and responsibility |
| SQL geo-replication / failover groups | Aurora Global Database, RDS cross-Region read replicas | Aurora Global promotes a secondary in about a minute (AWS's stated design) ⚠ VERIFY |
| Cosmos DB multi-region writes | DynamoDB global tables | Last-writer-wins by default; strongly consistent multi-Region mode has constraints (Active-active vs active-passive) |
| Traffic Manager / Front Door failover | Route 53 failover records, ARC routing controls, Global Accelerator | DNS-based failover is bounded by TTLs and client caches (Failover with Route 53) |
| ARM/Bicep to rebuild an environment | CloudFormation, CDK or Terraform in the DR Region | The same principle: if you cannot rebuild from code, you cannot recover quickly |
Architecture
The spectrum, with what is running in the recovery Region at each point:
A warm-standby failover for the Accounts API, as a sequence (the numbers in the runbook are your own measurements):
Security-boundary view: the DR Region should live in a separate account with its own KMS keys (multi-Region keys or per-Region copies with a re-encryption step), its own IAM roles, and an SCP that restricts who can modify the replication and backup configuration. Backups in the isolated account should be immutable (Vault Lock) so an attacker with primary-account credentials cannot delete them.
Step-by-step implementation
This is a design exercise, with the output being a decision record and a timed runbook.
List the business services and their tolerances
For each important business service, record the impact tolerance, the RTO and RPO (from Availability vocabulary), the data classification and any residency constraint. Put this in a table. If you cannot fill a cell, you have found the first thing to resolve.
Ask what each strategy can honestly deliver
Use DrStrategySelector.Recommend(rto, rpo) from the lab library as a first filter. It returns the cheapest strategy whose illustrative planning RTO and RPO fit, or no strategy if the target is out of reach. Then replace the planning numbers with measured ones as you test.
Check the dependency chain
For the chosen strategy, list every dependency the service has: identity provider, secrets, KMS keys, DNS, container registry, third-party APIs, schemes. Each must exist, or be reachable, in the recovery Region. The weakest one sets the real RTO.
Write the runbook and time it on paper
List each step with an owner and a duration estimate. Use FailoverBudget.Total and Slack against the RTO. Include detection and decision time, not just technical steps. A negative slack means you must change the design or the target.
Prepare the recovery Region with code
Create the DR stack from the same IaC with a "mode" parameter (off, scaled down, full). Raise quotas. Copy AMIs and container images (ECR replication). Replicate secrets and keys as appropriate.
Rehearse, measure and update
Run a tabletop exercise first, then an actual failover in a non-production environment. Record measured RTO and RPO, fix the gaps, repeat on a schedule. Update the planning table with measured values.
AWS Console instructions
Design lesson, with a few places to look at the pieces:
- Backups and copies: Console → AWS Backup → Backup plans / Backup vaults / Jobs to see plans, vaults and cross-Region copy jobs.
- Replication status: Console → RDS → Databases → Global database (Aurora), DynamoDB → Tables → Global tables, S3 → bucket → Management → Replication rules.
- Recovery-Region quotas: Console → Service Quotas, switch the Region selector to the recovery Region and review vCPU, Fargate, Elastic IP and Lambda concurrency.
- Elastic Disaster Recovery: Console → AWS Elastic Disaster Recovery for source servers, replication status and launch settings ⚠ VERIFY availability and prices in your Region.
AWS CLI
PROFILE=learn-sandbox
PRIMARY=us-east-1
DR=us-west-2
# Compare service quotas between Regions (here: Fargate on-demand vCPU), because DR needs the same ceiling.
# Replace the quota code with the one that applies to you; list codes with list-service-quotas.
aws service-quotas list-service-quotas --service-code fargate --region $PRIMARY --profile $PROFILE \
--query "Quotas[].{Name:QuotaName,Value:Value}" --output table
aws service-quotas list-service-quotas --service-code fargate --region $DR --profile $PROFILE \
--query "Quotas[].{Name:QuotaName,Value:Value}" --output table
# Backup and restore: copy a recovery point from the primary vault to a vault in the DR Region (illustrative IDs).
aws backup start-copy-job \
--recovery-point-arn arn:aws:backup:us-east-1:111122223333:recovery-point:EXAMPLE \
--source-backup-vault-name hl-primary-vault \
--destination-backup-vault-arn arn:aws:backup:us-west-2:111122223333:backup-vault:hl-dr-vault \
--iam-role-arn arn:aws:iam::111122223333:role/hl-backup-role \
--profile $PROFILE --region $PRIMARY
# Check data-replication lag (Aurora Global Database example metric; names ⚠ VERIFY for your engine).
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name AuroraGlobalDBReplicationLag \
--dimensions Name=DBClusterIdentifier,Value=hl-dr-cluster --start-time "$(date -u -d '-15 minutes' +%FT%TZ)" \
--end-time "$(date -u +%FT%TZ)" --period 60 --statistics Maximum --profile $PROFILE --region $DRThe ARNs, vault names and cluster names are examples; this lesson deliberately does not create cross-Region resources, because multi-Region databases are not free.
Infrastructure as Code
The key idea is one stack, two modes. An environment parameter selects the strategy's footprint. This excerpt is illustrative (it is not part of the compiled lab) and shows the pattern; the lab's real stack takes MinTasks and MaxTasks the same way.
public enum DrMode { Primary, PilotLight, WarmStandby }
public sealed class AccountApiProps : StackProps
{
public DrMode Mode { get; init; } = DrMode.Primary;
}
// In the stack constructor:
var desiredTasks = props.Mode switch
{
DrMode.Primary => 18, // full capacity: peak 12, three AZs, AZ-loss headroom
DrMode.WarmStandby => 6, // running and taking smoke traffic, scaled up on failover
DrMode.PilotLight => 0, // deployed but off; the failover runbook sets it to 18
_ => throw new ArgumentOutOfRangeException(nameof(props), props.Mode, null),
};
var service = new FargateService(this, "Service", new FargateServiceProps
{
Cluster = cluster,
TaskDefinition = taskDefinition,
DesiredCount = desiredTasks,
});
// Deploy the same code to both Regions:
// new AccountApiStack(app, "hl-api-use1", new AccountApiProps { Mode = DrMode.Primary, Env = Use1 });
// new AccountApiStack(app, "hl-api-usw2", new AccountApiProps { Mode = DrMode.PilotLight, Env = Usw2 });Complete code
The lab's strategy selector, a worst-case RPO helper and a runbook timer: the arithmetic of this lesson, covered by PlanningTests.
namespace Harbourline.Resilience;
public enum DrStrategy { BackupAndRestore, PilotLight, WarmStandby, ActiveActive }
/// <summary>ILLUSTRATIVE planning values. Teaching heuristics, not AWS guarantees. Replace with measured numbers.</summary>
public sealed record StrategyProfile(DrStrategy Strategy, TimeSpan PlanningRto, TimeSpan PlanningRpo, double RelativeCost);
public static class DrStrategySelector
{
public static IReadOnlyList<StrategyProfile> Profiles { get; } =
[
new(DrStrategy.BackupAndRestore, TimeSpan.FromHours(8), TimeSpan.FromHours(4), 1.1),
new(DrStrategy.PilotLight, TimeSpan.FromHours(1), TimeSpan.FromMinutes(5), 1.3),
new(DrStrategy.WarmStandby, TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(1), 1.6),
new(DrStrategy.ActiveActive, TimeSpan.FromMinutes(1), TimeSpan.FromSeconds(5), 2.2),
];
public static StrategyProfile? Recommend(TimeSpan rto, TimeSpan rpo) =>
Profiles.Where(p => p.PlanningRto <= rto && p.PlanningRpo <= rpo)
.OrderBy(p => p.RelativeCost)
.FirstOrDefault();
// Disaster just before the next backup, and the latest copy has not reached the other Region yet.
public static TimeSpan WorstCaseRpo(TimeSpan backupInterval, TimeSpan copyLag) => backupInterval + copyLag;
}using Harbourline.Resilience;
(string Service, TimeSpan Rto, TimeSpan Rpo)[] services =
[
("Card authorisation", TimeSpan.FromMinutes(1), TimeSpan.FromSeconds(5)),
("Online banking", TimeSpan.FromMinutes(30), TimeSpan.FromMinutes(5)),
("Statements", TimeSpan.FromHours(12), TimeSpan.FromHours(6)),
("Faster Payments hub", TimeSpan.FromSeconds(10), TimeSpan.Zero),
];
foreach (var (service, rto, rpo) in services)
{
var pick = DrStrategySelector.Recommend(rto, rpo);
Console.WriteLine(pick is null
? $"{service,-22} RTO {rto,-9} RPO {rpo,-9} -> NO strategy fits: renegotiate or redesign"
: $"{service,-22} RTO {rto,-9} RPO {rpo,-9} -> {pick.Strategy} (about {pick.RelativeCost:0.0}x primary cost)");
}
// Backup-based worst-case RPO: hourly backups, 15 minutes to copy across Regions.
Console.WriteLine($"Worst-case RPO: {DrStrategySelector.WorstCaseRpo(TimeSpan.FromHours(1), TimeSpan.FromMinutes(15))}");Expected output:
Card authorisation RTO 00:01:00 RPO 00:00:05 -> ActiveActive (about 2.2x primary cost)
Online banking RTO 00:30:00 RPO 00:05:00 -> WarmStandby (about 1.6x primary cost)
Statements RTO 12:00:00 RPO 06:00:00 -> BackupAndRestore (about 1.1x primary cost)
Faster Payments hub RTO 00:00:10 RPO 00:00:00 -> NO strategy fits: renegotiate or redesign
Worst-case RPO: 01:15:00Code explained line by line
StrategyProfilerecords one strategy's planning RTO, RPO and a relative cost. The comment on it matters: these are heuristics for reasoning, not facts to quote in a regulator-facing document.Profilesis ordered cheapest first and every step is both faster and more expensive, whichPlanningTests.Profiles_get_more_expensive_as_they_get_fasterasserts.Recommendfilters to strategies whose planning RTO and RPO are no worse than the targets and returns the cheapest. Returningnullis deliberate: an unachievable target is a finding, not something to paper over.- The "Online banking" row asks for RTO 30 minutes and RPO 5 minutes. Pilot light's planning RTO is 1 hour, which does not fit, so warm standby is chosen. This is the module project's target.
- "Faster Payments hub" asks for 10 seconds and zero data loss: nothing fits. In a real bank that conversation is about a different architecture (for example single-writer ledger with synchronous replication inside a Region, plus a tested business-continuity process) and about what the scheme itself guarantees.
WorstCaseRpoadds the backup interval and the copy lag: with hourly backups and a 15-minute copy, a disaster just before the next backup loses up to 75 minutes.
Security considerations
- DR is part of the attack surface. Replicated data and promoted standby systems hold production data. Apply the same encryption (KMS keys in the recovery Region), access control, logging and vulnerability management. A neglected DR Region is where audits and attackers find things.
- Isolate backups from the primary's credentials. Cross-account copies plus Vault Lock protect against ransomware and malicious deletion, which replication does not.
- Break-glass and failover permissions. The ability to promote a database or change routing is powerful. Give it to named roles, require MFA, log every action to CloudTrail in a protected account, and rehearse using the same roles.
- Keys and secrets must exist in the recovery Region. Plan multi-Region KMS keys or re-encryption, and secret replication; test that the DR copy can decrypt its own data.
- Residency. Choosing a second Region for UK customer data is a legal question as well as a technical one (UK GDPR and data residency); involve the data protection officer.
⚠ Learning shortcut — not for production
This lesson keeps everything on paper, with us-west-2 as an example DR Region and illustrative cost multiples. A real design uses the Regions your legal and compliance teams approve, real quotes from the Pricing Calculator, and a signed-off business impact analysis. Nothing here makes a service resilient until it has been rehearsed, and nothing here is advice about regulatory compliance.
Banking use case
Harbourline's resilience committee assigns tiers and strategies. All numbers are fictional.
| Service | Tier | RTO / RPO | Strategy | Why |
|---|---|---|---|---|
| Card authorisation | 0 | 1 minute / seconds | Active-active across two Regions, partitioned by card range | Customers and merchants cannot wait; the bank accepts the complexity |
| Online and mobile banking, Accounts API | 1 | 30 min / 5 min | Warm standby with Aurora Global Database | Meets the targets at about 1.6 times cost; the module project |
| Notifications | 2 | 4 hours / 1 hour | Pilot light (queues replicated, compute off) | Customers tolerate delayed alerts |
| Statements | 2 | 12 hours / 6 hours | Backup and restore | The data is regenerable and the cost saving is significant |
| Reporting warehouse | 3 | 24 hours / 24 hours | Backup and restore | Business can wait |
The committee also records the chosen drill per service: Region-loss game day for tier 0, a failover drill for tier 1, and a restore test for tiers 2 and 3.
Common mistakes
- One strategy for everything. Either the whole estate is overpriced or critical services miss their tolerance.
- Calling replication "backup". It copies mistakes and deletions instantly.
- A DR Region you cannot scale into. Quotas, instance availability and image copies were never checked.
- An environment that was never rebuilt from code. Manual drift means a restore finds a different world.
- A runbook that begins with a control-plane action. Create the environment before the disaster.
- Forgetting identity, DNS, secrets and certificates. They are dependencies too, with their own Region stories.
- No failback plan. Getting back is another project.
- Never testing. An untested strategy is an assumption with a price tag.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Restore in the DR Region takes much longer than planned | Large data volume, sequential steps, missing parallelism | Time each step in a drill; restore in parallel; use continuous replication for the largest stores |
| Pilot light cannot scale to full size | Quotas in the DR Region too low | Raise quotas ahead of time; track them in the readiness checklist |
| Replicated data is corrupt | Corruption replicated before detection | Use point-in-time recovery in the DR Region; keep backups of replicated data |
| Failover works but customers cannot sign in | Identity provider or secrets only in the primary Region | Replicate or provide a recovery-Region path for identity and secrets |
| Application cannot decrypt DR data | KMS key not available or re-encrypted incorrectly | Use multi-Region keys or tested re-encryption; test decryption monthly |
| DNS still points at the failed Region for minutes | Long TTLs and client caches | Lower TTLs for failover records; test with the clients you have (Failover with Route 53) |
| Drill passes but a real event fails | Drill used the control plane, the console or special permissions | Rehearse with the real failover roles and data-plane actions |
Cost and cleanup
This lesson costs nothing. The scratch console and the lab tests run locally; the CLI commands in the "AWS CLI" section only read quotas (and the copy and metric commands are examples you do not need to run).
If you do build a real second Region later, expect: continuous data replication and live data stores for pilot light and warm standby, running compute for warm standby, and everything twice for active-active. Do not leave cross-Region replication running after an exercise: Aurora Global Database and DynamoDB global tables bill continuously (⚠ VERIFY current prices in the Pricing Calculator).
# Nothing was created. Confirm no lab resources are tagged anywhere you looked.
for r in us-east-1 us-west-2; do
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region $r --query "ResourceTagMappingList[].ResourceARN"
doneExercises
- 🟢 Guided: Run
PlanningTests, then the scratch console. Add a fifth service, "Internal HR portal" (RTO 2 days, RPO 1 day), and predict its strategy before running. - 🟡 Challenge: Harbourline's Payments Service has RTO 15 minutes, RPO 1 minute, and a monthly budget that allows 1.7 times the primary run cost. Choose a strategy, list the five dependencies that must exist in the recovery Region, and write the first eight runbook steps with estimated minutes. Compute the slack.
- 🔴 Live problem: "Our DR plan says warm standby, but last week's tabletop found that the DR Region's Fargate quota is 10 percent of production, the container images are only in the primary Region's registry, and the database replica has been paused for a month." Triage the three findings, explain the risk of each in business terms, and propose an owner, a fix and a date for each.
Assignment
Produce a DR strategy decision record for one Harbourline service in your portfolio repository: tolerances, chosen strategy, rejected alternatives and why, the dependency list, a timed runbook table, the cost multiple, and the test schedule. Use docs/dr-plan-template.md from the lab as the skeleton.
Solutions
RTO 2 days and RPO 1 day fit backup and restore (planning RTO 8 hours, RPO 4 hours) comfortably, and it is the cheapest option, so Recommend returns BackupAndRestore. The learning point: a lower tier should not be forced into a more expensive strategy to look consistent, but it still needs a tested restore.
RTO 15 minutes and RPO 1 minute match warm standby's planning values (15 minutes, 1 minute) exactly, at 1.6 times cost, within the 1.7 times budget. Dependencies that must exist in the recovery Region: the database replica (Aurora Global), container images (ECR replication), secrets and KMS keys, DNS and health checks, and the identity provider path. Runbook example: detect and confirm 3 min; decide 3 min; freeze deployments 1 min; promote database 2 min; scale compute 4 min; switch traffic 2 min (DNS TTL permitting); verify 3 min; notify 2 min. Total 20 minutes against a 15-minute RTO gives slack of minus 5, so the design needs changes: automate detection and decision (pre-agreed criteria), pre-scale the DR service to cover more of the load (hot-standby-like), and shorten the DNS TTL or use a data-plane routing control. The exercise shows how quickly "warm standby" stops meeting a 15-minute RTO once humans are in the loop.
Quota at 10 percent: the DR Region cannot scale to production; the business risk is a failover that only partly restores capacity, so request and verify increases now (owner: platform, this week). Images only in the primary registry: new tasks cannot start in the DR Region; the risk is a failed recovery; enable ECR cross-Region replication and add a drill step that pulls from the DR registry (owner: platform, this week). Replica paused for a month: the RPO is effectively a month, a severe breach of tolerance; resume replication, find out why it was paused, alarm on replication lag, and document the pause approval process (owner: database lead, immediately). Also add these to a DR readiness checklist reviewed weekly.
Knowledge check
Answer: In pilot light, data is replicated and core infrastructure exists but compute is off or not deployed, so it cannot serve requests until started and scaled. In warm standby, a scaled-down but fully functional copy is already running and can serve traffic at reduced capacity immediately; you only scale it up.
Answer: Replication copies data changes, including corruption, deletions and ransomware, to the second Region almost immediately. Backups with point-in-time recovery (ideally immutable and isolated) let you restore to a point before the damage.
Answer: Scaling is a control-plane operation, so it takes time and may be degraded during a large event. Pre-provisioned capacity in the recovery Region removes that dependency, at higher cost.
Answer: Multi-site active-active. Its difficulty is data consistency and write conflicts across Regions, plus the highest cost and operational complexity.
Answer: Unverified readiness of the recovery Region: low service quotas, missing images, keys or secrets, or an environment that cannot be created from code. These are all preparation failures, which is why drills matter.
Interview questions
- "Walk me through the four DR strategies." Backup and restore (data only, rebuild from code, hours), pilot light (replicated data, compute off, tens of minutes to an hour), warm standby (scaled-down working copy, minutes), active-active (full capacity in both Regions, near zero). Cost and complexity rise as recovery time falls.
- "How would you choose between warm standby and active-active?" By the cost of downtime versus the cost and complexity of multi-Region writes. If an RTO of minutes is acceptable, warm standby is far simpler. If seconds are required and the data model allows partitioned or conflict-tolerant writes, active-active.
- "What is the most common reason a DR test fails?" Missing dependencies in the recovery Region (quotas, images, secrets, keys, identity) and manual steps nobody timed.
- "Should failover be automatic?" Often not for a Region: false alarms cost availability and data, so many banks use fully scripted but human-approved failover, with automatic failover reserved for AZ-level events and for carefully tuned health checks.
Certification practice
A company needs to recover a workload within about 30 minutes after a Regional failure, with minimal data loss, and wants to avoid paying for a full-capacity second environment. Which strategy best fits?
A. Backup and restore. B. Pilot light with compute off. C. Warm standby with a scaled-down environment running in the recovery Region. D. Multi-site active-active at full capacity.
Answer: C. A warm standby serves traffic at reduced capacity immediately and scales up in minutes, at lower cost than active-active. A is too slow, B may exceed the RTO because compute must be started and deployed, and D exceeds the cost goal.
An operations team replicates an Aurora database to a second Region. After a faulty script deleted rows in the primary, the deleted data was also gone in the second Region. What should they have in place?
A. A larger secondary instance. B. Point-in-time recovery or backups that allow restoring to a time before the deletion. C. More replicas. D. Cross-zone load balancing.
Answer: B. Replication propagates deletions. Backups or point-in-time recovery are the protection against data disasters.
A company runs a pilot-light DR design. During a failover test, the recovery team cannot launch enough compute and fails the RTO. The cause is traced to default service quotas in the recovery Region. Which change best prevents recurrence?
A. Move the database to the primary Region only. B. Request and verify quota increases in the recovery Region ahead of time and add a quota check to the DR readiness checklist. C. Disable automatic scaling. D. Increase the backup frequency.
Answer: B. Quotas are per Region and must be raised before the event. A removes DR. C and D do not address launch capacity.
Completion checklist
- I can explain each strategy and what runs in the recovery Region
- I can say why pilot light and warm standby differ, and why backups are needed in every strategy
- I ran
PlanningTestsand the scratch console - I produced a decision record with a timed runbook for one service
- I confirmed no cross-Region resources were created or left running
Key takeaways
- The strategies differ in how much of the workload is already running in the recovery Region: cost rises as RTO and RPO shrink.
- Choose per business service, not per company, and prove the choice in a drill.
- Replication protects against losing a place; backups protect against losing correctness. Every strategy needs both.
- Preparation decides the outcome: infrastructure as code, quotas, images, keys, secrets, identity and a separate account for the recovery copy.
- Keep failover on the data plane and rehearse failback.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Resilience patterns in .NET: timeouts, retries, breakers, bulkheads and load shedding
Make a .NET API survive a slow or failing dependency with a layered resilience pipeline, a retry budget, a circuit breaker, a bulkhead, load shedding and a graceful fallback, and prove each layer with a fault-injecting test backend.
Active-active vs active-passive: consistency, conflicts and single-writer ledgers
Understand what changes when two Regions both accept writes: replication lag, conflict resolution, last-writer-wins, strong consistency, and why most banking ledgers stay single-writer with a hot or warm second Region.