Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Lab: Resilient Account API (resilience pipeline, multi-AZ scaling and a DR plan)

Prove a .NET balance API survives a flaky dependency with a fault-injecting test backend, optionally deploy a three-AZ ECS service with target-tracking scaling and drill an AZ failure, then size a warm-standby DR plan for an RTO of 30 minutes and an RPO of 5 minutes.

advanced⏱ 3 h 🟡 Low costSAA-C03SOA-C03DOP-C02

Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 · AWS SDK for .NET v4 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lab you can:

  • Run a layered resilience pipeline against a fault-injecting backend and explain what each of the 86 automated tests proves.
  • Read the resilience demo's output and tell, from the numbers, which layer (retry, budget, breaker, timeout) acted.
  • Compute the capacity needed to lose an Availability Zone without scaling out, and verify a scaling configuration with a report.
  • Deploy (optionally) a three-AZ ECS service behind an ALB with target-tracking scaling using either CDK or Terraform, break one AZ's worth of tasks on purpose, and measure the impact.
  • Complete a DR plan for the module project: choose a strategy for an RTO of 30 minutes and an RPO of 5 minutes, write the failover and failback runbook, and check it fits with FailoverBudget.
  • Tear everything down and prove nothing tagged Project=cea-lab remains.

This lab covers Lab 12.1 and Lab 12.2 from the module plan and the design half of the module project. Backup restore tests, FIS experiments and the tabletop drill have their own labs and lessons (AWS Backup in depth, Chaos engineering with FIS, Resilience Hub and game days).

Prerequisites

Why this matters in banking

A resilience design that has never been exercised is a claim, not a control. Regulators and auditors ask for evidence: tests that fire, measurements from drills, and a plan with timed steps. This lab produces that evidence for one small service, so that you learn the habit: every resilience statement in a design document should point at a test result, a measurement or a drill record.

The idea in plain language

The lab has four parts that build on each other. Parts 1 to 3 run locally and cost nothing. Parts 4 and 5 deploy for under an hour. Part 6 is a paper exercise that turns what you measured into a plan.

PartWhat you doCostTime
1Run the tests and the resilience demofree20 min
2Experiment with the pipeline and the load shedderfree30 min
3Capacity and scaling arithmetic, and the scaling report logicfree20 min
4Deploy the multi-AZ ECS service (CDK or Terraform)credits30 min
5Scale test and AZ-failure drillcredits30 min
6DR plan for RTO 30 min and RPO 5 minfree40 min
7Teardown and verificationfree10 min

Analogy: parts 1 to 3 are the fire-safety inspection of a new branch on paper and in a simulator. Parts 4 and 5 are the actual fire drill with the doors open. Part 6 is the evacuation plan you pin to the wall, with the timings you measured.

What the AWS services do here

  • Amazon ECS on Fargate runs three or more copies of the API (tasks) across three Availability Zones.
  • Application Load Balancer sends requests to healthy tasks in all three AZs and drains tasks cleanly.
  • Application Auto Scaling keeps the task count between 3 and 9 using CPU and requests-per-target targets.
  • Amazon CloudWatch raises an alarm when a target has been unhealthy for three minutes, and records the metrics you use to measure the drill.
  • Amazon Route 53 (optional) adds a health check and a failover PRIMARY record if you own a hosted zone.

Nothing in this lab needs a database, so there is no Aurora or DynamoDB to pay for.

Microsoft / Azure comparison

You know (Microsoft/Azure)This lab usesWhat's different
xUnit tests with a fake HttpMessageHandlerxUnit v3 with ScriptedBackendThe same idea, packaged as a reusable fault injector that scripts failures call by call
Polly policies in an ASP.NET appMicrosoft.Extensions.Http.Resilience pipelinePipelines compose strategies in a fixed, reviewable order
App Service plan scale-out across zonesECS service across three AZ subnets with target trackingYou choose subnets per AZ and the minimum task count
Chaos Studio experimentA manual "stop tasks in one AZ" drill (the FIS lesson automates this)Doing it by hand first teaches what to measure
Azure Load TestingA simple curl loop (or any load tool you like)Keep the load tiny: this is a lab, not a benchmark

Architecture

The failure you will create in Part 5:

Step-by-step implementation

Part 1: run the tests and the demo (free)

Open the lab

From the repository root, the lab is labs/m12-resilient-account-api. Look at its README.md and the folder layout: src (library, demo, tools), tests, infra/cdk, infra/terraform, docs.

Run the tests in Docker

MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
  -v "$PWD/labs/m12-resilient-account-api:/src" \
  -v cea-nuget-m12:/root/.nuget/packages \
  -w /src mcr.microsoft.com/dotnet/sdk:10.0 \
  sh -c "dotnet test --solution ResilientAccountApi.slnx; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"

On PowerShell, use a backtick for line continuation and ${PWD} for the path. The first run downloads NuGet packages into the named volume and takes a few minutes; later runs take about a minute. Expected: Test run summary: Passed! and total: 86. dotnet test does not compile the CDK project; run dotnet build ResilientAccountApi.slnx to prove the infrastructure code compiles.

Run the demo

MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
  -v "$PWD/labs/m12-resilient-account-api:/src" -v cea-nuget-m12:/root/.nuget/packages -w /src \
  mcr.microsoft.com/dotnet/sdk:10.0 \
  sh -c "dotnet run --project src/Harbourline.Resilience.Demo; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"

You should see four scenarios. Compare with the sample in Resilience patterns in .NET. Timings vary, but the shape must match: one backend call for a healthy service, three for two blips, a timeout after about three seconds for the brown-out, and backend calls shrinking from 3 to 2 to 1 and then zero when the circuit opens.

Part 2: experiment with the pipeline (free)

Change one number at a time

Edit src/Harbourline.Resilience.Demo/Program.cs. Change RetryBudget to null (no budget), run the demo, and look at the "Hard outage" scenario: how many backend calls does each of the first calls make now? Put the budget back. Then raise BreakerMinimumThroughput to 20 and see when the circuit opens.

Write a test for a new rule

Add a test to tests/Harbourline.Resilience.Tests/ResiliencePipelineTests.cs that proves a 429 response is retried and the retried call succeeds (hint: new ScriptedBackend(Fault.Error((HttpStatusCode)429), Fault.Ok())). Run the tests again.

Break the load shedder on purpose

In LoadSheddingTests, change the expectation of Best_effort_traffic_is_refused_first so it fails, read the failure message, then restore it. Understanding why a test fails is part of trusting it.

Part 3: capacity arithmetic (free)

Compute AZ-loss capacity

For a peak of 12 tasks, work out the total tasks needed to survive the loss of one AZ with 2, 3 and 4 AZs. Check against ScalingTests (Total_tasks_to_survive_one_az_loss).

Apply the scaling report rules

Read src/Harbourline.Resilience.Tools/ScalingReport.cs. Which of these settings does it flag, and why: min 2 max 10 in 3 AZs; min 3 max 3; min 4 max 12 in 3 AZs; min 3 max 9?

Part 4: deploy the multi-AZ service (credits, under 1 hour)

Choose one tool. Do not deploy both: the stacks use the same lab tags and similar names.

Sign in and check the account

aws sso login --profile learn-sandbox
aws sts get-caller-identity --profile learn-sandbox

Confirm the account number is your sandbox, not any other.

Preview, then deploy

cd labs/m12-resilient-account-api/infra/cdk
cdk diff   -c owner=<your-name> --profile learn-sandbox
cdk deploy -c owner=<your-name> --profile learn-sandbox

Read the plan or diff before you approve: you should see a VPC, three subnets, an ALB, a target group, an ECS cluster and service, a task definition, scaling targets and policies, a log group and an alarm. Nothing else.

Wait for health and call the service

# Replace with the ALB DNS name from the stack outputs (output AlbDnsName / alb_dns_name).
ALB=<alb-dns-name>
PROFILE=learn-sandbox; REGION=us-east-1

# Find the load balancer and its target group from the DNS name (works for both the CDK and Terraform stacks).
ALB_ARN=$(aws elbv2 describe-load-balancers --profile $PROFILE --region $REGION \
  --query "LoadBalancers[?DNSName=='$ALB'].LoadBalancerArn | [0]" --output text)
TG_ARN=$(aws elbv2 describe-target-groups --load-balancer-arn "$ALB_ARN" --profile $PROFILE --region $REGION \
  --query "TargetGroups[0].TargetGroupArn" --output text)

aws elbv2 describe-target-health --target-group-arn "$TG_ARN" --profile $PROFILE --region $REGION \
  --query "TargetHealthDescriptions[].{AZ:Target.AvailabilityZone,State:TargetHealth.State}" --output table
curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://$ALB/

Wait until three targets are healthy, one per AZ. If tasks stay unhealthy, see the troubleshooting table.

Part 5: scale test and AZ-failure drill (credits)

Start a steady request loop (keep it light)

# One request every 0.5 seconds; prints the HTTP status and time. Leave it running in a second terminal.
while true; do
  curl -s -o /dev/null -m 3 -w "$(date +%T) %{http_code} %{time_total}s\n" http://$ALB/
  sleep 0.5
done

All lines should be 200. This is your baseline.

Watch scaling (optional)

The placeholder container is not CPU hungry, so the lab will not scale much on its own. To see scaling, you can lower the CPU target to 10 percent in the stack (CDK TargetUtilizationPercent, Terraform target_value) and redeploy, or simply read the scaling configuration:

aws application-autoscaling describe-scalable-targets --service-namespace ecs \
  --profile learn-sandbox --region us-east-1
aws application-autoscaling describe-scaling-policies --service-namespace ecs \
  --profile learn-sandbox --region us-east-1

Use the scaling report from the Tools project (scaling 3) and confirm it reports no problems for your service.

Fail one AZ's tasks

CLUSTER=hl-m12-account-api   # Terraform name; for CDK use the ClusterName stack output
# List tasks with their AZs.
for t in $(aws ecs list-tasks --cluster $CLUSTER --query "taskArns[]" --output text --profile learn-sandbox --region us-east-1); do
  aws ecs describe-tasks --cluster $CLUSTER --tasks $t --query "tasks[0].[availabilityZone,taskArn]" --output text --profile learn-sandbox --region us-east-1
done

# Stop every task that runs in ONE AZ (replace us-east-1a with the AZ you choose).
for t in $(aws ecs list-tasks --cluster $CLUSTER --query "taskArns[]" --output text --profile learn-sandbox --region us-east-1); do
  az=$(aws ecs describe-tasks --cluster $CLUSTER --tasks $t --query "tasks[0].availabilityZone" --output text --profile learn-sandbox --region us-east-1)
  [ "$az" = "us-east-1a" ] && aws ecs stop-task --cluster $CLUSTER --task $t --reason "Lab 12 AZ drill" --profile learn-sandbox --region us-east-1
done

Watch the request loop. Because the other two AZs still have tasks and the ALB removes stopped tasks, you should see no errors or a very short blip. ECS then launches replacements and AZ rebalancing restores an even spread. Write down: number of non-200 responses, the time until three targets were healthy again, and which AZ each replacement landed in.

This drill stops tasks, not the AZ itself: it models the loss of capacity in one AZ, which is what the capacity formula protects against. A full AZ-impairment test needs Fault Injection Service (Chaos engineering with FIS).

Practise a zonal shift (optional, read the ELB lesson first)

# ALB_ARN was set in Part 4 (looked up from the DNS name).
aws arc-zonal-shift start-zonal-shift --resource-identifier "$ALB_ARN" --away-from use1-az1 \
  --expires-in 15m --comment "Lab 12 practice" --profile learn-sandbox --region us-east-1
# Watch the loop and target health for a few minutes, then cancel.
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile learn-sandbox --region us-east-1

⚠ VERIFY the command, the --expires-in format and that the ALB is eligible in your account. For the Terraform stack the name is hl-m12-account-api; for CDK find the generated name first. Cancel the shift (cancel-zonal-shift --zonal-shift-id <id>) before you tear down.

Part 6: the DR plan for the module project (free)

State the targets

Copy labs/m12-resilient-account-api/docs/dr-plan-template.md to your own repository. The module project targets are RTO 30 minutes and RPO 5 minutes. Fill the targets table.

Choose and justify a strategy

Use DrStrategySelector.Recommend (see PlanningTests) as a first filter: for 30 minutes and 5 minutes it returns warm standby. Write a paragraph: why not pilot light, why not active-active, and what would make you change your mind (for example a tighter RTO or a regulator-set tolerance).

Write the runbook with times

Fill the failover table. Use your Part 5 measurements for any step you timed (for example "traffic stable after losing capacity: 40 s") and honest estimates for the rest. Typical steps for a Regional failover: detect and confirm, decide and record, freeze deployments, fence the old primary, promote the database, scale the DR service, switch traffic (health check or routing control), verify with synthetic transactions, communicate.

Check it against the RTO

Sum the minutes with FailoverBudget.Total and compute Slack against 30 minutes (see Runbook_slack_goes_negative_when_the_steps_do_not_fit_the_rto in PlanningTests). If the slack is negative, change the design: pre-provision more DR capacity, automate a step, shorten detection, or renegotiate the target. Record what you changed.

Write the failback section and the drill schedule

Failback is a planned change: replicate in the opposite direction, verify, switch in a quiet window. Add a quarterly drill and a note of who signs off the measured RTO and RPO.

Part 7: teardown and verification

Do this the same day, even if you stopped part-way.

# Cancel any zonal shift
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile learn-sandbox --region us-east-1

# CDK
cd labs/m12-resilient-account-api/infra/cdk && cdk destroy -c owner=<your-name> --profile learn-sandbox
# or Terraform
cd labs/m12-resilient-account-api/infra/terraform && terraform destroy

# Nothing tagged for this lab should remain (the tagging API can lag a few minutes).
aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
  --profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"

The last command should print [].

AWS Console instructions

  • ECS: Console → ECS → Clusters → hl-m12-account-api → Services → api → Tasks (task AZ in the details), Events (what ECS did), Service auto scaling.
  • ALB: Console → EC2 → Load Balancers (the lab ALB, its AZs), Target Groups → Targets (health by AZ and reason codes).
  • CloudWatch: Console → CloudWatch → Alarms (hl-m12-account-api-unhealthy-targets in Terraform; a generated name in CDK), Metrics → ApplicationELB (HealthyHostCount, UnHealthyHostCount, RequestCount, HTTPCode_Target_5XX_Count).
  • Route 53 (optional): Console → Route 53 → Health checks and your hosted zone's failover record.

AWS CLI

The commands for each part are inline above. The most useful repeat commands:

# Is every AZ represented among healthy targets?
aws elbv2 describe-target-health --target-group-arn <arn> --profile learn-sandbox --region us-east-1 \
  --query "TargetHealthDescriptions[].{AZ:Target.AvailabilityZone,State:TargetHealth.State,Reason:TargetHealth.Reason}" --output table

# What did ECS do, and when?
aws ecs describe-services --cluster hl-m12-account-api --services api --profile learn-sandbox --region us-east-1 \
  --query "services[0].events[:10].[createdAt,message]" --output table

# What did auto scaling do?
aws application-autoscaling describe-scaling-activities --service-namespace ecs --max-results 10 \
  --profile learn-sandbox --region us-east-1

Infrastructure as Code

The stacks are in infra/cdk and infra/terraform and implement the same design. The decisions worth reading in the code:

DecisionCDK (C#)Terraform
Three AZs, no NATMaxAzs = 3, NatGateways = 0three aws_subnet over slice(azs, 0, 3), no NAT resource
Tasks spread, rebalancedAvailabilityZoneRebalancing.ENABLEDavailability_zone_rebalancing = "ENABLED"
Bad deployments roll backCircuitBreaker = new DeploymentCircuitBreaker { Rollback = true }deployment_circuit_breaker { enable, rollback }
Clean drainingDeregistrationDelay = 30 s, StopTimeout = 35 sderegistration_delay = 30, stopTimeout = 35
ScalingScaleOnCpuUtilization, ScaleOnRequestCount, scale-in 5 mintwo aws_appautoscaling_policy target-tracking policies
RoutingLEAST_OUTSTANDING_REQUESTSload_balancing_algorithm_type = "least_outstanding_requests"
TagsTags.Of(app).Add(...) for the four mandatory tagsprovider default_tags
Log retentionRetentionDays.ONE_WEEKretention_in_days = 7

terraform validate passes on the Terraform stack (checked with Terraform 1.17), and the CDK project compiles as part of the solution build. cdk synth and cdk-nag were not run in this lab (the README lists adding cdk-nag as an exercise).

Complete code

The code is in the lab folder. Browse it in this order:

  1. src/Harbourline.Resilience/HarbourlineResilience.cs and ResilienceOptions.cs: the pipeline.
  2. src/Harbourline.Resilience/ScriptedBackend.cs: the fault injector.
  3. src/Harbourline.Resilience/LoadShedder.cs and LoadSheddingMiddleware.cs: priority shedding.
  4. src/Harbourline.Resilience/BalanceService.cs: the stale-balance fallback.
  5. src/Harbourline.Resilience/ReadinessGate.cs, AzAwareness.cs, CapacityPlanner.cs: the Multi-AZ helpers.
  6. src/Harbourline.Resilience/DrStrategySelector.cs, FailoverBudget.cs, Route53HealthModel.cs, EpochFence.cs, ReplicationModels.cs: the DR arithmetic and models.
  7. src/Harbourline.Resilience.Tools: read-only reports for scaling and Route 53 health checks.
  8. tests/Harbourline.Resilience.Tests: 86 tests, none of which needs an AWS account.

The test file map, so you know what proves what:

Test fileWhat it provesLesson
PlanningTestsNines arithmetic, strategy choice, DNS and runbook timing1, 6, 8
ScalingTestsAZ-loss capacity, scaling-report rules2
ReadinessGateTestsDraining on SIGTERM3
AzAwarenessTestsAZ discovery and spread checking4
ResiliencePipelineTestsRetry, timeouts, breaker open and close, bulkhead, retry budget, POST never retried5
LoadSheddingTestsPriority shedding and the 503 middleware5
BalanceServiceTestsStale fallback, and 404 not hidden5
ReplicationModelTestsLast-writer-wins loses updates; ledger can overdraw; single writer holds7
Route53HealthModelTests, EpochFenceTests18 percent rule, detection and TTL maths, fencing8

Code explained line by line

A short tour of the three most important ideas in the code:

  1. Pipeline order is the design. In HarbourlineResilience.cs the strategies are added bulkhead, total timeout, retry, breaker, attempt timeout. Swap the total timeout and the retry and the The_total_timeout_caps_the_sum_of_all_retries test fails, which is the quickest way to feel why the order matters.
  2. The fault injector is deterministic. ScriptedBackend plays call 1, call 2 and so on from a script, and repeats the last step. Tests assert backend.Calls, so "the circuit is open" becomes an arithmetic fact: calls to the backend stop increasing.
  3. Nothing in the tests uses AWS. The tools that do call AWS (Harbourline.Resilience.Tools) hold no logic: they read data and pass it to pure functions (ScalingReport.Analyse, Route53HealthModel) that the tests cover. That is the pattern to copy for your own tooling.

Security considerations

  • Sandbox only. The stack opens an HTTP listener to the internet and gives tasks public IPs. It is deliberately minimal and short-lived.
  • No secrets. There are none in the lab; keep it that way. Credentials come from the SSO profile.
  • Tags and cleanup are controls. The four mandatory tags make leftover resources findable; the teardown check proves they are gone.
  • Do not point the lab at production resources. The AZ drill stops tasks. Run it only against the lab cluster.
  • Placeholder image. The default container is a public image. Pin a digest or use your own ECR image in anything beyond this lab.

⚠ Learning shortcut — not for production

Public subnets with public task IPs (no NAT gateway), an HTTP-only ALB open to the internet, local Terraform state, a placeholder container and no WAF. The resilience pipeline's fallback cache is in memory on one process. Production needs HTTPS and WAF, private subnets with VPC endpoints, remote state, a shared cache with a maximum age, measured thresholds, and the DR plan reviewed by the service owner, risk and the operational-resilience team. Nothing in this lab makes a system compliant with any regulation.

Banking use case

The lab models Harbourline's Accounts API in the module project. The deliverables map to what a service owner would show an internal audit or a regulator-facing resilience review:

EvidenceWhere it comes from
"The API survives a slow or failing Core Banking"ResiliencePipelineTests, BalanceServiceTests, demo output
"Losing an AZ does not need a scale-out"CapacityPlanner arithmetic, Part 5 drill measurements
"We know our scaling is sane"scaling report output
"The DR strategy fits the RTO and RPO"DrStrategySelector result, completed dr-plan-template.md, FailoverBudget slack
"Split brain is prevented"the fencing section of the runbook, EpochFenceTests

All names and numbers are fictional.

Common mistakes

  • Deploying both CDK and Terraform. They collide on names and tags. Use one.
  • Forgetting -c owner= (CDK) or owner in terraform.tfvars: the Owner tag is empty and cleanup checks lose precision.
  • Using the placeholder image and an unrealistic health path. The default nginx image answers /; your own API should use /health/ready.
  • Running the AZ drill on an ECS service with desiredCount below 3. There may be no task left in the other AZs.
  • Running the Docker test command from the wrong folder. The -v path must point at the lab folder.
  • Leaving the stack up overnight. About USD 0.10 per hour becomes a few dollars; set a reminder.
  • Treating the planning numbers as facts. DrStrategySelector values are illustrative.

Troubleshooting

SymptomLikely causeFix
Docker run fails with a path error on WindowsMSYS path conversion or wrong quotingUse MSYS_NO_PATHCONV=1 in Git Bash, or PowerShell with ${PWD}; check the folder exists
First test run is very slowNuGet restore of CDK and AWS SDK packagesWait; later runs reuse the cea-nuget-m12 volume
Tests fail on a slow machine (timing)Real timers in a few resilience testsRe-run once; keep Docker memory at 3 GB or more; do not run other heavy jobs
Tasks never become healthyHealth path or port does not match the imageSet healthCheckPath and containerPort for your image
Tasks stuck in PENDING ("cannot pull image")Image tag missing or wrong architecture (the task is ARM64)Use an arm64 image, or change the CPU architecture in the stack
terraform apply fails on a name that existsEarlier attempt left resourcesRun teardown first, then apply again
Zonal shift command not found or rejectedOlder CLI, or ALB not eligibleUpdate the CLI; check ARC supported resources; skip the optional step
describe-target-health shows only two AZsA task is unhealthy or stoppedWait for ECS to replace it; look at service events
Tagging API still lists resources after destroyIndex lagWait a few minutes and run the check again; confirm in the console

Cost and cleanup

  • Parts 1 to 3 and 6: free.
  • Parts 4 and 5: about USD 0.10 per hour in total for three small Fargate tasks and one ALB, plus pennies for logs ⚠ VERIFY current prices in the AWS Pricing Calculator. A one-hour session costs well under one dollar. If the service scales to its maximum (9 tasks) the cost rises accordingly for as long as it runs.
  • Optional Route 53 health check and hosted zone: small monthly charges; delete them the same day.

Teardown commands are in Part 7. Afterwards the tagged-resources query must print []. If it does not, delete the listed resources by hand and run it again.

Exercises

  • 🟢 Guided: Complete Parts 1, 3 and 6 without deploying anything. Submit the completed dr-plan-template.md and the output of the test run.
  • 🟡 Challenge: Add a new strategy of your own to the pipeline: a hedged request (send a second attempt after a short delay if the first is slow) using AddHedging for idempotent reads, with a test proving the second attempt can win. Explain the trade-off in extra load.
  • 🔴 Live problem: "Finance asks why the DR Region costs 60 percent of production when it serves no customers." Using DrStrategySelector profiles and your own measured recovery steps, prepare a one-page options paper: keep warm standby, move to pilot light with a longer RTO, or move to backup and restore, with the RTO and RPO each option delivers, the risk, and a recommendation the service owner can sign.

Assignment

Publish labs/m12-resilient-account-api (or your fork) to your portfolio repository with: the test output, your Part 5 measurements table (baseline latency, errors during the drill, recovery time), the completed DR plan, and a one-page summary of which resilience claims are now backed by evidence and which are still assumptions.

Solutions

Knowledge check

Interview questions

  • "Show me evidence that this API is resilient." Open the test suite: each layer has a test, the demo shows the numbers, the drill measurements show AZ-loss behaviour, and the DR plan has timed steps with slack against the RTO.
  • "How did you test failure without an AWS account?" A deterministic fault-injecting HTTP handler that scripts failures by call number, plus pure functions for the arithmetic, so the tests run anywhere.
  • "What did the AZ drill teach you?" That static stability (capacity already running in the surviving AZs) carries the first minutes; scaling and replacement only restore the margin afterwards.

Certification practice

Completion checklist

  • I ran the test suite in Docker and saw Passed! with 86 tests
  • I ran the demo and can explain each row of its output
  • I computed AZ-loss capacity and read the scaling report rules
  • (Optional) I deployed one stack, ran the AZ drill and recorded measurements
  • I completed the DR plan with timed runbook steps and computed the slack
  • I ran the teardown and the tagged-resources check printed []

Key takeaways

  • Every resilience claim should point at a test, a measurement or a drill.
  • A deterministic fault injector turns "the circuit opens" into an assertion on a counter.
  • Static stability carries the first minutes of an AZ loss; capacity per AZ is a calculation, not a feeling.
  • A DR plan is a table of timed steps checked against the RTO, and the first draft usually does not fit.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page