Lab: Resilient Account API (resilience pipeline, multi-AZ scaling and a DR plan)
Prove a .NET balance API survives a flaky dependency with a fault-injecting test backend, optionally deploy a three-AZ ECS service with target-tracking scaling and drill an AZ failure, then size a warm-standby DR plan for an RTO of 30 minutes and an RPO of 5 minutes.
Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Run a layered resilience pipeline against a fault-injecting backend and explain what each of the 86 automated tests proves.
- Read the resilience demo's output and tell, from the numbers, which layer (retry, budget, breaker, timeout) acted.
- Compute the capacity needed to lose an Availability Zone without scaling out, and verify a scaling configuration with a report.
- Deploy (optionally) a three-AZ ECS service behind an ALB with target-tracking scaling using either CDK or Terraform, break one AZ's worth of tasks on purpose, and measure the impact.
- Complete a DR plan for the module project: choose a strategy for an RTO of 30 minutes and an RPO of 5 minutes, write the failover and failback runbook, and check it fits with
FailoverBudget. - Tear everything down and prove nothing tagged
Project=cea-labremains.
This lab covers Lab 12.1 and Lab 12.2 from the module plan and the design half of the module project. Backup restore tests, FIS experiments and the tabletop drill have their own labs and lessons (AWS Backup in depth, Chaos engineering with FIS, Resilience Hub and game days).
Prerequisites
- Lessons 1 to 8 of this module: Availability vocabulary, Auto Scaling everywhere, ELB for resilience, Multi-AZ by default, Resilience patterns in .NET, The four DR strategies, Active-active vs active-passive, Failover with Route 53
- Docker Desktop. The .NET 10 SDK is optional: every command in this lab runs in a container.
- For the optional deployment (Parts 4 and 5): AWS CLI v2 with the
learn-sandboxSSO profile, and either the AWS CDK CLI or Terraform 1.10 or later. Use only the sandbox account.
Why this matters in banking
A resilience design that has never been exercised is a claim, not a control. Regulators and auditors ask for evidence: tests that fire, measurements from drills, and a plan with timed steps. This lab produces that evidence for one small service, so that you learn the habit: every resilience statement in a design document should point at a test result, a measurement or a drill record.
The idea in plain language
The lab has four parts that build on each other. Parts 1 to 3 run locally and cost nothing. Parts 4 and 5 deploy for under an hour. Part 6 is a paper exercise that turns what you measured into a plan.
| Part | What you do | Cost | Time |
|---|---|---|---|
| 1 | Run the tests and the resilience demo | free | 20 min |
| 2 | Experiment with the pipeline and the load shedder | free | 30 min |
| 3 | Capacity and scaling arithmetic, and the scaling report logic | free | 20 min |
| 4 | Deploy the multi-AZ ECS service (CDK or Terraform) | credits | 30 min |
| 5 | Scale test and AZ-failure drill | credits | 30 min |
| 6 | DR plan for RTO 30 min and RPO 5 min | free | 40 min |
| 7 | Teardown and verification | free | 10 min |
Analogy: parts 1 to 3 are the fire-safety inspection of a new branch on paper and in a simulator. Parts 4 and 5 are the actual fire drill with the doors open. Part 6 is the evacuation plan you pin to the wall, with the timings you measured.
What the AWS services do here
- Amazon ECS on Fargate runs three or more copies of the API (tasks) across three Availability Zones.
- Application Load Balancer sends requests to healthy tasks in all three AZs and drains tasks cleanly.
- Application Auto Scaling keeps the task count between 3 and 9 using CPU and requests-per-target targets.
- Amazon CloudWatch raises an alarm when a target has been unhealthy for three minutes, and records the metrics you use to measure the drill.
- Amazon Route 53 (optional) adds a health check and a failover PRIMARY record if you own a hosted zone.
Nothing in this lab needs a database, so there is no Aurora or DynamoDB to pay for.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | This lab uses | What's different |
|---|---|---|
xUnit tests with a fake HttpMessageHandler | xUnit v3 with ScriptedBackend | The same idea, packaged as a reusable fault injector that scripts failures call by call |
| Polly policies in an ASP.NET app | Microsoft.Extensions.Http.Resilience pipeline | Pipelines compose strategies in a fixed, reviewable order |
| App Service plan scale-out across zones | ECS service across three AZ subnets with target tracking | You choose subnets per AZ and the minimum task count |
| Chaos Studio experiment | A manual "stop tasks in one AZ" drill (the FIS lesson automates this) | Doing it by hand first teaches what to measure |
| Azure Load Testing | A simple curl loop (or any load tool you like) | Keep the load tiny: this is a lab, not a benchmark |
Architecture
The failure you will create in Part 5:
Step-by-step implementation
Part 1: run the tests and the demo (free)
Open the lab
From the repository root, the lab is labs/m12-resilient-account-api. Look at its README.md and the folder layout: src (library, demo, tools), tests, infra/cdk, infra/terraform, docs.
Run the tests in Docker
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
-v "$PWD/labs/m12-resilient-account-api:/src" \
-v cea-nuget-m12:/root/.nuget/packages \
-w /src mcr.microsoft.com/dotnet/sdk:10.0 \
sh -c "dotnet test --solution ResilientAccountApi.slnx; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"On PowerShell, use a backtick for line continuation and ${PWD} for the path. The first run downloads NuGet packages into the named volume and takes a few minutes; later runs take about a minute. Expected: Test run summary: Passed! and total: 86. dotnet test does not compile the CDK project; run dotnet build ResilientAccountApi.slnx to prove the infrastructure code compiles.
Run the demo
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
-v "$PWD/labs/m12-resilient-account-api:/src" -v cea-nuget-m12:/root/.nuget/packages -w /src \
mcr.microsoft.com/dotnet/sdk:10.0 \
sh -c "dotnet run --project src/Harbourline.Resilience.Demo; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"You should see four scenarios. Compare with the sample in Resilience patterns in .NET. Timings vary, but the shape must match: one backend call for a healthy service, three for two blips, a timeout after about three seconds for the brown-out, and backend calls shrinking from 3 to 2 to 1 and then zero when the circuit opens.
Part 2: experiment with the pipeline (free)
Change one number at a time
Edit src/Harbourline.Resilience.Demo/Program.cs. Change RetryBudget to null (no budget), run the demo, and look at the "Hard outage" scenario: how many backend calls does each of the first calls make now? Put the budget back. Then raise BreakerMinimumThroughput to 20 and see when the circuit opens.
Write a test for a new rule
Add a test to tests/Harbourline.Resilience.Tests/ResiliencePipelineTests.cs that proves a 429 response is retried and the retried call succeeds (hint: new ScriptedBackend(Fault.Error((HttpStatusCode)429), Fault.Ok())). Run the tests again.
Break the load shedder on purpose
In LoadSheddingTests, change the expectation of Best_effort_traffic_is_refused_first so it fails, read the failure message, then restore it. Understanding why a test fails is part of trusting it.
Part 3: capacity arithmetic (free)
Compute AZ-loss capacity
For a peak of 12 tasks, work out the total tasks needed to survive the loss of one AZ with 2, 3 and 4 AZs. Check against ScalingTests (Total_tasks_to_survive_one_az_loss).
Apply the scaling report rules
Read src/Harbourline.Resilience.Tools/ScalingReport.cs. Which of these settings does it flag, and why: min 2 max 10 in 3 AZs; min 3 max 3; min 4 max 12 in 3 AZs; min 3 max 9?
Part 4: deploy the multi-AZ service (credits, under 1 hour)
Choose one tool. Do not deploy both: the stacks use the same lab tags and similar names.
Sign in and check the account
aws sso login --profile learn-sandbox
aws sts get-caller-identity --profile learn-sandboxConfirm the account number is your sandbox, not any other.
Preview, then deploy
cd labs/m12-resilient-account-api/infra/cdk
cdk diff -c owner=<your-name> --profile learn-sandbox
cdk deploy -c owner=<your-name> --profile learn-sandboxRead the plan or diff before you approve: you should see a VPC, three subnets, an ALB, a target group, an ECS cluster and service, a task definition, scaling targets and policies, a log group and an alarm. Nothing else.
Wait for health and call the service
# Replace with the ALB DNS name from the stack outputs (output AlbDnsName / alb_dns_name).
ALB=<alb-dns-name>
PROFILE=learn-sandbox; REGION=us-east-1
# Find the load balancer and its target group from the DNS name (works for both the CDK and Terraform stacks).
ALB_ARN=$(aws elbv2 describe-load-balancers --profile $PROFILE --region $REGION \
--query "LoadBalancers[?DNSName=='$ALB'].LoadBalancerArn | [0]" --output text)
TG_ARN=$(aws elbv2 describe-target-groups --load-balancer-arn "$ALB_ARN" --profile $PROFILE --region $REGION \
--query "TargetGroups[0].TargetGroupArn" --output text)
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" --profile $PROFILE --region $REGION \
--query "TargetHealthDescriptions[].{AZ:Target.AvailabilityZone,State:TargetHealth.State}" --output table
curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://$ALB/Wait until three targets are healthy, one per AZ. If tasks stay unhealthy, see the troubleshooting table.
Part 5: scale test and AZ-failure drill (credits)
Start a steady request loop (keep it light)
# One request every 0.5 seconds; prints the HTTP status and time. Leave it running in a second terminal.
while true; do
curl -s -o /dev/null -m 3 -w "$(date +%T) %{http_code} %{time_total}s\n" http://$ALB/
sleep 0.5
doneAll lines should be 200. This is your baseline.
Watch scaling (optional)
The placeholder container is not CPU hungry, so the lab will not scale much on its own. To see scaling, you can lower the CPU target to 10 percent in the stack (CDK TargetUtilizationPercent, Terraform target_value) and redeploy, or simply read the scaling configuration:
aws application-autoscaling describe-scalable-targets --service-namespace ecs \
--profile learn-sandbox --region us-east-1
aws application-autoscaling describe-scaling-policies --service-namespace ecs \
--profile learn-sandbox --region us-east-1Use the scaling report from the Tools project (scaling 3) and confirm it reports no problems for your service.
Fail one AZ's tasks
CLUSTER=hl-m12-account-api # Terraform name; for CDK use the ClusterName stack output
# List tasks with their AZs.
for t in $(aws ecs list-tasks --cluster $CLUSTER --query "taskArns[]" --output text --profile learn-sandbox --region us-east-1); do
aws ecs describe-tasks --cluster $CLUSTER --tasks $t --query "tasks[0].[availabilityZone,taskArn]" --output text --profile learn-sandbox --region us-east-1
done
# Stop every task that runs in ONE AZ (replace us-east-1a with the AZ you choose).
for t in $(aws ecs list-tasks --cluster $CLUSTER --query "taskArns[]" --output text --profile learn-sandbox --region us-east-1); do
az=$(aws ecs describe-tasks --cluster $CLUSTER --tasks $t --query "tasks[0].availabilityZone" --output text --profile learn-sandbox --region us-east-1)
[ "$az" = "us-east-1a" ] && aws ecs stop-task --cluster $CLUSTER --task $t --reason "Lab 12 AZ drill" --profile learn-sandbox --region us-east-1
doneWatch the request loop. Because the other two AZs still have tasks and the ALB removes stopped tasks, you should see no errors or a very short blip. ECS then launches replacements and AZ rebalancing restores an even spread. Write down: number of non-200 responses, the time until three targets were healthy again, and which AZ each replacement landed in.
This drill stops tasks, not the AZ itself: it models the loss of capacity in one AZ, which is what the capacity formula protects against. A full AZ-impairment test needs Fault Injection Service (Chaos engineering with FIS).
Practise a zonal shift (optional, read the ELB lesson first)
# ALB_ARN was set in Part 4 (looked up from the DNS name).
aws arc-zonal-shift start-zonal-shift --resource-identifier "$ALB_ARN" --away-from use1-az1 \
--expires-in 15m --comment "Lab 12 practice" --profile learn-sandbox --region us-east-1
# Watch the loop and target health for a few minutes, then cancel.
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile learn-sandbox --region us-east-1⚠ VERIFY the command, the --expires-in format and that the ALB is eligible in your account. For the Terraform stack the name is hl-m12-account-api; for CDK find the generated name first. Cancel the shift (cancel-zonal-shift --zonal-shift-id <id>) before you tear down.
Part 6: the DR plan for the module project (free)
State the targets
Copy labs/m12-resilient-account-api/docs/dr-plan-template.md to your own repository. The module project targets are RTO 30 minutes and RPO 5 minutes. Fill the targets table.
Choose and justify a strategy
Use DrStrategySelector.Recommend (see PlanningTests) as a first filter: for 30 minutes and 5 minutes it returns warm standby. Write a paragraph: why not pilot light, why not active-active, and what would make you change your mind (for example a tighter RTO or a regulator-set tolerance).
Write the runbook with times
Fill the failover table. Use your Part 5 measurements for any step you timed (for example "traffic stable after losing capacity: 40 s") and honest estimates for the rest. Typical steps for a Regional failover: detect and confirm, decide and record, freeze deployments, fence the old primary, promote the database, scale the DR service, switch traffic (health check or routing control), verify with synthetic transactions, communicate.
Check it against the RTO
Sum the minutes with FailoverBudget.Total and compute Slack against 30 minutes (see Runbook_slack_goes_negative_when_the_steps_do_not_fit_the_rto in PlanningTests). If the slack is negative, change the design: pre-provision more DR capacity, automate a step, shorten detection, or renegotiate the target. Record what you changed.
Write the failback section and the drill schedule
Failback is a planned change: replicate in the opposite direction, verify, switch in a quiet window. Add a quarterly drill and a note of who signs off the measured RTO and RPO.
Part 7: teardown and verification
Do this the same day, even if you stopped part-way.
# Cancel any zonal shift
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile learn-sandbox --region us-east-1
# CDK
cd labs/m12-resilient-account-api/infra/cdk && cdk destroy -c owner=<your-name> --profile learn-sandbox
# or Terraform
cd labs/m12-resilient-account-api/infra/terraform && terraform destroy
# Nothing tagged for this lab should remain (the tagging API can lag a few minutes).
aws resourcegroupstaggingapi get-resources \
--tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
--profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"The last command should print [].
AWS Console instructions
- ECS: Console → ECS → Clusters → hl-m12-account-api → Services → api → Tasks (task AZ in the details), Events (what ECS did), Service auto scaling.
- ALB: Console → EC2 → Load Balancers (the lab ALB, its AZs), Target Groups → Targets (health by AZ and reason codes).
- CloudWatch: Console → CloudWatch → Alarms (
hl-m12-account-api-unhealthy-targetsin Terraform; a generated name in CDK), Metrics → ApplicationELB (HealthyHostCount,UnHealthyHostCount,RequestCount,HTTPCode_Target_5XX_Count). - Route 53 (optional): Console → Route 53 → Health checks and your hosted zone's failover record.
AWS CLI
The commands for each part are inline above. The most useful repeat commands:
# Is every AZ represented among healthy targets?
aws elbv2 describe-target-health --target-group-arn <arn> --profile learn-sandbox --region us-east-1 \
--query "TargetHealthDescriptions[].{AZ:Target.AvailabilityZone,State:TargetHealth.State,Reason:TargetHealth.Reason}" --output table
# What did ECS do, and when?
aws ecs describe-services --cluster hl-m12-account-api --services api --profile learn-sandbox --region us-east-1 \
--query "services[0].events[:10].[createdAt,message]" --output table
# What did auto scaling do?
aws application-autoscaling describe-scaling-activities --service-namespace ecs --max-results 10 \
--profile learn-sandbox --region us-east-1Infrastructure as Code
The stacks are in infra/cdk and infra/terraform and implement the same design. The decisions worth reading in the code:
| Decision | CDK (C#) | Terraform |
|---|---|---|
| Three AZs, no NAT | MaxAzs = 3, NatGateways = 0 | three aws_subnet over slice(azs, 0, 3), no NAT resource |
| Tasks spread, rebalanced | AvailabilityZoneRebalancing.ENABLED | availability_zone_rebalancing = "ENABLED" |
| Bad deployments roll back | CircuitBreaker = new DeploymentCircuitBreaker { Rollback = true } | deployment_circuit_breaker { enable, rollback } |
| Clean draining | DeregistrationDelay = 30 s, StopTimeout = 35 s | deregistration_delay = 30, stopTimeout = 35 |
| Scaling | ScaleOnCpuUtilization, ScaleOnRequestCount, scale-in 5 min | two aws_appautoscaling_policy target-tracking policies |
| Routing | LEAST_OUTSTANDING_REQUESTS | load_balancing_algorithm_type = "least_outstanding_requests" |
| Tags | Tags.Of(app).Add(...) for the four mandatory tags | provider default_tags |
| Log retention | RetentionDays.ONE_WEEK | retention_in_days = 7 |
terraform validate passes on the Terraform stack (checked with Terraform 1.17), and the CDK project compiles as part of the solution build. cdk synth and cdk-nag were not run in this lab (the README lists adding cdk-nag as an exercise).
Complete code
The code is in the lab folder. Browse it in this order:
src/Harbourline.Resilience/HarbourlineResilience.csandResilienceOptions.cs: the pipeline.src/Harbourline.Resilience/ScriptedBackend.cs: the fault injector.src/Harbourline.Resilience/LoadShedder.csandLoadSheddingMiddleware.cs: priority shedding.src/Harbourline.Resilience/BalanceService.cs: the stale-balance fallback.src/Harbourline.Resilience/ReadinessGate.cs,AzAwareness.cs,CapacityPlanner.cs: the Multi-AZ helpers.src/Harbourline.Resilience/DrStrategySelector.cs,FailoverBudget.cs,Route53HealthModel.cs,EpochFence.cs,ReplicationModels.cs: the DR arithmetic and models.src/Harbourline.Resilience.Tools: read-only reports for scaling and Route 53 health checks.tests/Harbourline.Resilience.Tests: 86 tests, none of which needs an AWS account.
The test file map, so you know what proves what:
| Test file | What it proves | Lesson |
|---|---|---|
PlanningTests | Nines arithmetic, strategy choice, DNS and runbook timing | 1, 6, 8 |
ScalingTests | AZ-loss capacity, scaling-report rules | 2 |
ReadinessGateTests | Draining on SIGTERM | 3 |
AzAwarenessTests | AZ discovery and spread checking | 4 |
ResiliencePipelineTests | Retry, timeouts, breaker open and close, bulkhead, retry budget, POST never retried | 5 |
LoadSheddingTests | Priority shedding and the 503 middleware | 5 |
BalanceServiceTests | Stale fallback, and 404 not hidden | 5 |
ReplicationModelTests | Last-writer-wins loses updates; ledger can overdraw; single writer holds | 7 |
Route53HealthModelTests, EpochFenceTests | 18 percent rule, detection and TTL maths, fencing | 8 |
Code explained line by line
A short tour of the three most important ideas in the code:
- Pipeline order is the design. In
HarbourlineResilience.csthe strategies are added bulkhead, total timeout, retry, breaker, attempt timeout. Swap the total timeout and the retry and theThe_total_timeout_caps_the_sum_of_all_retriestest fails, which is the quickest way to feel why the order matters. - The fault injector is deterministic.
ScriptedBackendplays call 1, call 2 and so on from a script, and repeats the last step. Tests assertbackend.Calls, so "the circuit is open" becomes an arithmetic fact: calls to the backend stop increasing. - Nothing in the tests uses AWS. The tools that do call AWS (
Harbourline.Resilience.Tools) hold no logic: they read data and pass it to pure functions (ScalingReport.Analyse,Route53HealthModel) that the tests cover. That is the pattern to copy for your own tooling.
Security considerations
- Sandbox only. The stack opens an HTTP listener to the internet and gives tasks public IPs. It is deliberately minimal and short-lived.
- No secrets. There are none in the lab; keep it that way. Credentials come from the SSO profile.
- Tags and cleanup are controls. The four mandatory tags make leftover resources findable; the teardown check proves they are gone.
- Do not point the lab at production resources. The AZ drill stops tasks. Run it only against the lab cluster.
- Placeholder image. The default container is a public image. Pin a digest or use your own ECR image in anything beyond this lab.
⚠ Learning shortcut — not for production
Public subnets with public task IPs (no NAT gateway), an HTTP-only ALB open to the internet, local Terraform state, a placeholder container and no WAF. The resilience pipeline's fallback cache is in memory on one process. Production needs HTTPS and WAF, private subnets with VPC endpoints, remote state, a shared cache with a maximum age, measured thresholds, and the DR plan reviewed by the service owner, risk and the operational-resilience team. Nothing in this lab makes a system compliant with any regulation.
Banking use case
The lab models Harbourline's Accounts API in the module project. The deliverables map to what a service owner would show an internal audit or a regulator-facing resilience review:
| Evidence | Where it comes from |
|---|---|
| "The API survives a slow or failing Core Banking" | ResiliencePipelineTests, BalanceServiceTests, demo output |
| "Losing an AZ does not need a scale-out" | CapacityPlanner arithmetic, Part 5 drill measurements |
| "We know our scaling is sane" | scaling report output |
| "The DR strategy fits the RTO and RPO" | DrStrategySelector result, completed dr-plan-template.md, FailoverBudget slack |
| "Split brain is prevented" | the fencing section of the runbook, EpochFenceTests |
All names and numbers are fictional.
Common mistakes
- Deploying both CDK and Terraform. They collide on names and tags. Use one.
- Forgetting
-c owner=(CDK) orownerinterraform.tfvars: the Owner tag is empty and cleanup checks lose precision. - Using the placeholder image and an unrealistic health path. The default nginx image answers
/; your own API should use/health/ready. - Running the AZ drill on an ECS service with
desiredCountbelow 3. There may be no task left in the other AZs. - Running the Docker test command from the wrong folder. The
-vpath must point at the lab folder. - Leaving the stack up overnight. About USD 0.10 per hour becomes a few dollars; set a reminder.
- Treating the planning numbers as facts.
DrStrategySelectorvalues are illustrative.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Docker run fails with a path error on Windows | MSYS path conversion or wrong quoting | Use MSYS_NO_PATHCONV=1 in Git Bash, or PowerShell with ${PWD}; check the folder exists |
| First test run is very slow | NuGet restore of CDK and AWS SDK packages | Wait; later runs reuse the cea-nuget-m12 volume |
| Tests fail on a slow machine (timing) | Real timers in a few resilience tests | Re-run once; keep Docker memory at 3 GB or more; do not run other heavy jobs |
| Tasks never become healthy | Health path or port does not match the image | Set healthCheckPath and containerPort for your image |
Tasks stuck in PENDING ("cannot pull image") | Image tag missing or wrong architecture (the task is ARM64) | Use an arm64 image, or change the CPU architecture in the stack |
terraform apply fails on a name that exists | Earlier attempt left resources | Run teardown first, then apply again |
| Zonal shift command not found or rejected | Older CLI, or ALB not eligible | Update the CLI; check ARC supported resources; skip the optional step |
describe-target-health shows only two AZs | A task is unhealthy or stopped | Wait for ECS to replace it; look at service events |
| Tagging API still lists resources after destroy | Index lag | Wait a few minutes and run the check again; confirm in the console |
Cost and cleanup
- Parts 1 to 3 and 6: free.
- Parts 4 and 5: about USD 0.10 per hour in total for three small Fargate tasks and one ALB, plus pennies for logs ⚠ VERIFY current prices in the AWS Pricing Calculator. A one-hour session costs well under one dollar. If the service scales to its maximum (9 tasks) the cost rises accordingly for as long as it runs.
- Optional Route 53 health check and hosted zone: small monthly charges; delete them the same day.
Teardown commands are in Part 7. Afterwards the tagged-resources query must print []. If it does not, delete the listed resources by hand and run it again.
Exercises
- 🟢 Guided: Complete Parts 1, 3 and 6 without deploying anything. Submit the completed
dr-plan-template.mdand the output of the test run. - 🟡 Challenge: Add a new strategy of your own to the pipeline: a hedged request (send a second attempt after a short delay if the first is slow) using
AddHedgingfor idempotent reads, with a test proving the second attempt can win. Explain the trade-off in extra load. - 🔴 Live problem: "Finance asks why the DR Region costs 60 percent of production when it serves no customers." Using
DrStrategySelectorprofiles and your own measured recovery steps, prepare a one-page options paper: keep warm standby, move to pilot light with a longer RTO, or move to backup and restore, with the RTO and RPO each option delivers, the risk, and a recommendation the service owner can sign.
Assignment
Publish labs/m12-resilient-account-api (or your fork) to your portfolio repository with: the test output, your Part 5 measurements table (baseline latency, errors during the drill, recovery time), the completed DR plan, and a one-page summary of which resilience claims are now backed by evidence and which are still assumptions.
Solutions
Without a budget the three scenarios that fail make a full set of retries every time: in the hard outage each call makes 3 backend calls until the circuit opens, so the circuit opens after two calls instead of three, and total load on the failing backend is higher. With the budget, retries run out and later calls make fewer attempts. That is the retry-storm protection visible in numbers.
[Fact]
public async Task A_429_is_retried_and_the_retry_succeeds()
{
var backend = new ScriptedBackend(Fault.Error((HttpStatusCode)429), Fault.Ok());
await using var provider = Build(backend, FastOptions());
var balance = await provider.GetRequiredService<CoreBankingClient>().GetBalanceAsync("ACC-0000000001");
Assert.Equal("GBP", balance.Currency);
Assert.Equal(2, backend.Calls);
}IsTransient treats 408, 429 and 5xx as retryable, so the test passes without changing the pipeline.
Min 2 max 10 in 3 AZs: a Problem (minimum below the AZ count). Min 3 max 3: a Warning (cannot add capacity). Min 4 max 12 in 3 AZs: Info only (4 does not divide evenly across 3 AZs). Min 3 max 9: no findings.
Detect and confirm 3 min, decide and record 3, freeze deployments 1, fence the old primary 2, promote the database 3, scale the DR service 5, switch traffic 2, verify with synthetic transactions 4, notify 2: total 25 minutes, slack against 30 is +5. If your measured database promotion takes 8 minutes, slack falls to 0; at 12 minutes it is negative and you must pre-scale, automate or renegotiate. The point is the conversation the number forces, not the number itself.
With AddHedging (HedgingStrategyOptions<HttpResponseMessage>) you set a Delay after which a second attempt starts in parallel and the first successful response wins. In a test, make call 1 Fault.Slow(2 s) and call 2 Fault.Ok(), set the hedging delay to 100 ms, and assert the result returns in well under 2 s with backend.Calls == 2. Trade-off: hedging cuts tail latency but increases load by up to the hedge rate, so restrict it to idempotent reads, add it under a retry budget, and never enable it when the dependency is already overloaded.
Warm standby: RTO about 15 to 30 minutes, RPO about a minute, cost about 1.6 times; meets the target. Pilot light: RTO about an hour, RPO minutes, about 1.3 times; saves roughly a fifth of the DR cost but breaches a 30-minute RTO. Backup and restore: RTO many hours, RPO hours, about 1.1 times; unacceptable for a tier 1 service. Recommendation: keep warm standby for the Accounts API, reduce idle cost by running the DR service at the minimum that passes health checks and smoke traffic, using Graviton and Savings Plans for the DR compute (⚠ VERIFY eligibility), and revisit if the service owner agrees a longer RTO. All figures are planning values to be replaced by measured and quoted numbers.
Knowledge check
Answer: The circuit breaker has opened after enough failures, so calls fail immediately with BrokenCircuitException without reaching the backend. This protects the dependency and frees your threads.
Answer: The two surviving AZs hold six tasks, and the ALB stops sending traffic to deregistered tasks, so the remaining capacity absorbs the load. This is static stability from the capacity formula.
Answer: Pre-provision more capacity in the recovery Region so the scale-up step shrinks, automate or parallelise slow steps (promotion, scaling), shorten detection and decision time with clear criteria, or renegotiate the RTO if the business accepts the risk.
Answer: A retried money-moving POST could duplicate a payment if the first attempt succeeded but the response was lost. Retrying is only safe for idempotent operations, or with an idempotency key the server de-duplicates.
Answer: None: the tagged-resources query for Project=cea-lab and Lab=m12-ha must return an empty list, and any optional Route 53 health check, hosted-zone records or zonal shifts you created must be deleted or cancelled.
Interview questions
- "Show me evidence that this API is resilient." Open the test suite: each layer has a test, the demo shows the numbers, the drill measurements show AZ-loss behaviour, and the DR plan has timed steps with slack against the RTO.
- "How did you test failure without an AWS account?" A deterministic fault-injecting HTTP handler that scripts failures by call number, plus pure functions for the arithmetic, so the tests run anywhere.
- "What did the AZ drill teach you?" That static stability (capacity already running in the surviving AZs) carries the first minutes; scaling and replacement only restore the margin afterwards.
Certification practice
An ECS service runs 9 tasks evenly across three Availability Zones behind an Application Load Balancer. One zone becomes unavailable. Assuming peak load needs 6 tasks, what is the expected effect and why?
A. The service is down until new tasks launch. B. Traffic continues to be served by the remaining 6 tasks with at most a brief blip, because capacity in the surviving zones already covers peak. C. The load balancer stops working. D. All tasks restart.
Answer: B. Three zones with three tasks each leave six tasks after losing a zone, which equals peak, so no scale-out is needed. The load balancer routes around the failed zone.
A deployment of a new task definition starts failing health checks. The team wants the ECS service to stop the rollout and return to the last working version automatically. Which setting provides this?
A. Increase the deregistration delay. B. Enable the ECS deployment circuit breaker with rollback. C. Turn off health checks. D. Add more subnets.
Answer: B. The deployment circuit breaker detects a failing rollout and rolls back to the last steady state. The other options do not stop or reverse a failing deployment.
After a drill, a service's Application Auto Scaling target shows minimum 2 and maximum 2 across three Availability Zones. What risk does this configuration create?
A. None; it is cost-efficient. B. It cannot scale out under load or recover capacity after losing an AZ, and two tasks cannot cover three zones. C. It will scale in to zero. D. It disables health checks.
Answer: B. Equal minimum and maximum disables scaling, and a minimum below the AZ count means some zones have no tasks. Capacity cannot be added when needed.
Completion checklist
- I ran the test suite in Docker and saw
Passed!with 86 tests - I ran the demo and can explain each row of its output
- I computed AZ-loss capacity and read the scaling report rules
- (Optional) I deployed one stack, ran the AZ drill and recorded measurements
- I completed the DR plan with timed runbook steps and computed the slack
- I ran the teardown and the tagged-resources check printed
[]
Key takeaways
- Every resilience claim should point at a test, a measurement or a drill.
- A deterministic fault injector turns "the circuit opens" into an assertion on a counter.
- Static stability carries the first minutes of an AZ loss; capacity per AZ is a calculation, not a feeling.
- A DR plan is a table of timed steps checked against the RTO, and the first draft usually does not fit.
Further reading
- Elastic Load Balancing for resilience and Failover with Route 53 in this module
- Module 12 overview — the module project and the other labs
- Disaster recovery options in the cloud (AWS whitepaper)
- Build resilient HTTP apps (Microsoft Learn)
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Failover with Route 53: health checks, failover routing, TTLs and split brain
Design DNS-based Regional failover with Route 53 health checks and failover or weighted records, calculate how long failover really takes, understand the control-plane trap and Application Recovery Controller routing controls, and prevent split brain.
Database failover across Regions
Choose and rehearse cross-Region database recovery: Aurora Global Database switchover and failover, DynamoDB global tables (MREC and MRSC), and RDS cross-Region read replica promotion, with a runbook and C# helpers that make the decision measurable.