Skip to Content
Cloud Engineering Academy
Projects

Capstone 5: Highly available banking application

A banking account service that survives a failed task, a lost Availability Zone and, with a rehearsed runbook, a lost Region: a three-tier three-AZ VPC, ALB, ECS Fargate with Auto Scaling, RDS PostgreSQL Multi-AZ, ElastiCache, AWS Backup with a cross-Region copy, alarms that page on symptoms, and a timed DR drill, built on the Module 12 lab plus Terraform for the stack.

advanced⏱ 10 h 🟡 Low costSAA-C03SOA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · Terraform 1.10+ with AWS provider 6.x · Amazon ECS on Fargate (arm64) · RDS for PostgreSQL · ElastiCache for Valkey · AWS Backup · AWS CLI v2 · Regions us-east-1 (primary) and us-west-2 (DR). Verified against official sources on 2026-11-18.

What you are building

An account service for the fictional Harbourline Bank that keeps answering when parts of it break. The stack:

  1. a VPC in three Availability Zones with three tiers: public (load balancer and NAT only), app (containers, no public IP) and data (a route table with no route to the internet);
  2. an Application Load Balancer in front of ECS Fargate tasks that Auto Scaling grows and shrinks, with the minimum set so that losing one AZ still leaves enough capacity;
  3. RDS for PostgreSQL Multi-AZ and ElastiCache for Valkey with a replica in another AZ, both encrypted, with RDS-managed credentials in Secrets Manager;
  4. AWS Backup with a daily plan and a cross-Region copy, plus an optional pilot-light replica in a DR Region;
  5. alarms that page on what customers feel (errors and latency) and only ticket on causes, with a dashboard;
  6. a DR drill: a timestamped event log turned into a measured RTO, RPO and a report by a small tested tool.

This capstone wraps the Module 12 lab (labs/m12-resilient-account-api: resilience patterns for .NET, scaling and DR arithmetic, a DR plan template, 86 tests), reuses the VPC and isolation pattern of the Module 2 lab and the observability patterns of the Module 11 lab, and adds the Terraform for the whole stack and the drill tool.

Educational project

The default container is an nginx placeholder, so the stack proves the infrastructure, not an application that uses the database and cache. Real systems add HTTPS with WAF, one NAT gateway per AZ, backups in a separate account with Vault Lock, and a DR plan agreed with the business. Nothing here makes a system compliant with any operational-resilience rule. All data is fictional.

Learning objectives

By the end you can:

  • Draw the three tiers and say which route table makes the data tier isolated, and which security-group references allow each hop.
  • Size the minimum task count so that the loss of one AZ leaves peak capacity, and justify it with the lab's capacity maths.
  • Explain what Multi-AZ protects against (an AZ or instance failure) and what it does not (a bad DELETE, a Region outage), and pick the control for each.
  • Restore from an AWS Backup recovery point and record how long it took.
  • Choose a DR strategy with a stated RTO and RPO and defend its cost, then measure the real numbers with a drill.
  • Design alarms so that failover that customers do not notice does not wake anyone up.
  • Deploy, break things on purpose, and tear down with a cost estimate.

The architecture

The repository adds the failover sequence and the security-boundary view in projects/p5-ha-banking-app/docs/architecture.md.

What each failure does

FailureWhat protects youNeeds a human?
One task crashesHealth check, ECS replaces itNo
One AZ unavailableTasks, standby database and cache replica in other AZs; ALB stops routing to the dead AZNo
A dependency (Core Banking) is slowThe lab's timeout, retry budget, circuit breaker and stale-balance fallbackNo
A bad deploymentECS deployment circuit breaker rolls backNo
Database corrupted or droppedPoint-in-time recovery, AWS Backup restoreYes
Whole Region downCross-Region copy or replica, runbook, Route 53Yes: decision, then runbook

Prerequisites and the lessons it builds on

What is in the repository

WhereWhat
labs/m12-resilient-account-apiThe resilience library (Polly pipeline, load shedding, readiness gate, AZ helpers, capacity and DR calculators), CDK and Terraform for a multi-AZ Fargate service, the DR plan template, 86 tests
labs/m02-secure-vpc, labs/m11-observe-banking-apiThe isolation pattern and the observability patterns, referenced
projects/p5-ha-banking-appThe capstone: README.md, checklist.md, docs/ (architecture, five ADRs, security controls, testing, monitoring, cost and teardown, DR drill guide), infra/terraform (network, data, compute, backup and DR, monitoring), the drill tool src/Harbourline.HaBanking.DrDrill with 18 tests, and drills/sample-drill.json

Build path

Run the tests (free). The m12 lab (86 tests) and the drill tool (18 tests) in Docker. Run the drill tool on the sample log and read the report it prints.

Do the tabletop drill (free). Walk the runbook in the DR plan template with the sample numbers, and write down where you think the time goes before you measure anything.

Deploy the stack. terraform apply in infra/terraform with your owner name; wait about 20 to 30 minutes for the Multi-AZ database. Open the ALB address and confirm three healthy targets across three AZs.

Break things in order. Stop a task, scale under load, restore a backup, fail the database over, fail the cache over, simulate an AZ loss. Record the result and time of each.

Add the DR layer and run the drill. Set enable_dr_replica = true, apply, run the game day from the DR drill guide, save the event log and produce the report.

Tear down both Regions, delete recovery points first, and prove nothing is left.

What was verified when this page was written

CheckResult
Lab m12 suite (Docker, .NET 10 SDK)86 passed, 0 failed
Drill tool suite18 passed, 0 failed; the sample drill reports Verdict Pass, RTO about 24 minutes, RPO 48 seconds
terraform validate on infra/terraform (Terraform 1.14, AWS provider 6.x), including the DR-Region provider aliasValid
Not runterraform apply, any AWS deployment, a restore, a failover or a real drill. Prices are estimates; engine and Valkey versions, CLI flags and the FIS and zonal-shift steps are marked ⚠ VERIFY in the repository docs

Security controls

The repository's docs/security-controls.md lists eleven controls with how to verify each. The ones that define this design:

ControlHow it is shown
Data tier is isolatedThe data route table has only the local route; Reachability Analyzer from the internet gateway to the database interface returns not reachable
Each hop is allowed by referenceALB to tasks to database and cache by security-group ID, never CIDR; the default security group allows nothing
Encryption everywhereCustomer-managed KMS key with rotation for the database, cache, logs and alarm topic; a separate key in the DR Region; TLS in transit
No stored passwordsRDS creates and rotates the master secret; the task execution role can read only that secret
Backups survive their sourceCross-Region copy with its own key (production adds a separate account and Vault Lock)
Safe deploymentsECS circuit breaker with rollback; deletion protection variable

Honest gaps: no WAF and an open ALB in the lab, one NAT gateway, backups in the same account, an empty task role because no real application is wired in, and no GuardDuty or Config here (the landing-zone capstone enables them organisation-wide).

Cost and teardown

Roughly USD 0.20 per hour while deployed (NAT gateway, ALB, three small Fargate tasks, a Multi-AZ database, two cache nodes, alarms and logs), about USD 145 a month if forgotten. A six-hour session is under USD 1.50. The optional DR replica adds about USD 0.02 per hour plus data transfer. Prices are list-price estimates dated 2026-11-18, not re-fetched from the pricing pages ⚠ VERIFY.

Teardown checklist

Delete the recovery points in both vaults (Terraform cannot destroy a vault that holds them), run terraform destroy, delete any manual snapshots and restored test instances, then run the tag query for Project=cea-lab and Lab=p5-ha-banking-app in both Regions. KMS keys sit in a 7-day pending-deletion state. The repository's docs/cost-and-teardown.md has the ordered commands.

Acceptance criteria

  • The m12 and drill-tool tests pass; the sample drill reports Pass.
  • Three targets are healthy across three AZs, and the data tier has no route to the internet.
  • You stopped a task, failed the database over and failed the cache over, with times recorded and no customer-visible errors beyond a short blip.
  • A backup was restored and the data checked; the restored instance was deleted.
  • The customer-impact alarm did not fire during the simulated AZ loss, and did fire when you broke the application.
  • A drill report exists with measured RTO and RPO and an action for every miss.
  • Both Regions are destroyed and nothing tagged for the project remains.

Portfolio-ready checklist

The full list is projects/p5-ha-banking-app/checklist.md. The essentials:

  • Five ADRs: three-tier three-AZ VPC, Multi-AZ database and cache, Fargate with target tracking, pilot light with cross-Region backup, page on symptoms.
  • Diagrams that match what you deployed, including the failover sequence.
  • Evidence you collected: the isolation proof, the failure-test table with times, the restore time, the drill report.
  • A written list of what production would still need: WAF, HTTPS, a NAT gateway per AZ, a separate backup account with Vault Lock, an agreed impact tolerance, a regulator-ready test schedule.
  • Extensions: wire in the Module 3 API with Npgsql and a cache-aside layer, a Synthetics canary, an alarm forwarder from the DR Region, Aurora Global Database as the managed cross-Region option, AWS Fault Injection Service experiments.

Exercises

🟢 Guided. Change drills/sample-drill.json so that the database promotion takes 25 minutes. Predict the verdict and the slowest step before you run the tool, then check.

🟡 Challenge. Add a HttpCode_ELB_5XX_Count alarm to the customer-impact composite alarm, and explain in one sentence the difference between an ELB 5xx and a target 5xx for a customer's experience.

🔴 Live problem. Ticket INC-2043: "After last night's Multi-AZ failover the Account API threw errors for eleven minutes although the database was back after ninety seconds. The alarms said everything was fine." Name three places the eleven minutes could have gone (think about connection pools, DNS caching and health checks) and what evidence you would collect for each.

Interview questions

  1. Your database is Multi-AZ. Does it need backups? Why?
  2. How would you prove your RTO to an auditor, and why is "the database was promoted" the wrong end of the stopwatch?
  3. What happens to in-flight requests during an RDS failover, and what must the application do about it?
  4. Why is the minimum task count tied to the number of Availability Zones?
  5. When is a pilot-light DR strategy the wrong choice?

Key takeaways

  • Availability is designed in layers: tasks, AZs, backups, Region. Each layer fails differently.
  • A standby is not a backup, and a backup is not a failover plan.
  • An RTO you have not timed is a guess; measure it with a drill and write down the misses.
  • Page on what customers feel; ticket on causes.
  • Always count the cost of the safety: NAT, Multi-AZ and a second Region are the bill.

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page