Capstone 5: Highly available banking application
A banking account service that survives a failed task, a lost Availability Zone and, with a rehearsed runbook, a lost Region: a three-tier three-AZ VPC, ALB, ECS Fargate with Auto Scaling, RDS PostgreSQL Multi-AZ, ElastiCache, AWS Backup with a cross-Region copy, alarms that page on symptoms, and a timed DR drill, built on the Module 12 lab plus Terraform for the stack.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · Terraform 1.10+ with AWS provider 6.x · Amazon ECS on Fargate (arm64) · RDS for PostgreSQL · ElastiCache for Valkey · AWS Backup · AWS CLI v2 · Regions us-east-1 (primary) and us-west-2 (DR). Verified against official sources on 2026-11-18.
What you are building
An account service for the fictional Harbourline Bank that keeps answering when parts of it break. The stack:
- a VPC in three Availability Zones with three tiers: public (load balancer and NAT only), app (containers, no public IP) and data (a route table with no route to the internet);
- an Application Load Balancer in front of ECS Fargate tasks that Auto Scaling grows and shrinks, with the minimum set so that losing one AZ still leaves enough capacity;
- RDS for PostgreSQL Multi-AZ and ElastiCache for Valkey with a replica in another AZ, both encrypted, with RDS-managed credentials in Secrets Manager;
- AWS Backup with a daily plan and a cross-Region copy, plus an optional pilot-light replica in a DR Region;
- alarms that page on what customers feel (errors and latency) and only ticket on causes, with a dashboard;
- a DR drill: a timestamped event log turned into a measured RTO, RPO and a report by a small tested tool.
This capstone wraps the Module 12 lab (labs/m12-resilient-account-api: resilience patterns for .NET, scaling and DR arithmetic, a DR plan template, 86 tests), reuses the VPC and isolation pattern of the Module 2 lab and the observability patterns of the Module 11 lab, and adds the Terraform for the whole stack and the drill tool.
Educational project
The default container is an nginx placeholder, so the stack proves the infrastructure, not an application that uses the database and cache. Real systems add HTTPS with WAF, one NAT gateway per AZ, backups in a separate account with Vault Lock, and a DR plan agreed with the business. Nothing here makes a system compliant with any operational-resilience rule. All data is fictional.
Learning objectives
By the end you can:
- Draw the three tiers and say which route table makes the data tier isolated, and which security-group references allow each hop.
- Size the minimum task count so that the loss of one AZ leaves peak capacity, and justify it with the lab's capacity maths.
- Explain what Multi-AZ protects against (an AZ or instance failure) and what it does not (a bad
DELETE, a Region outage), and pick the control for each. - Restore from an AWS Backup recovery point and record how long it took.
- Choose a DR strategy with a stated RTO and RPO and defend its cost, then measure the real numbers with a drill.
- Design alarms so that failover that customers do not notice does not wake anyone up.
- Deploy, break things on purpose, and tear down with a cost estimate.
The architecture
The repository adds the failover sequence and the security-boundary view in projects/p5-ha-banking-app/docs/architecture.md.
What each failure does
| Failure | What protects you | Needs a human? |
|---|---|---|
| One task crashes | Health check, ECS replaces it | No |
| One AZ unavailable | Tasks, standby database and cache replica in other AZs; ALB stops routing to the dead AZ | No |
| A dependency (Core Banking) is slow | The lab's timeout, retry budget, circuit breaker and stale-balance fallback | No |
| A bad deployment | ECS deployment circuit breaker rolls back | No |
| Database corrupted or dropped | Point-in-time recovery, AWS Backup restore | Yes |
| Whole Region down | Cross-Region copy or replica, runbook, Route 53 | Yes: decision, then runbook |
Prerequisites and the lessons it builds on
| You need | Lessons and labs |
|---|---|
| Networking | Module 2 overview, Public, private and isolated subnets, Security groups vs network ACLs, NAT gateway and egress, Elastic Load Balancing, VPC Flow Logs and Reachability Analyzer, Lab: prove it's private |
| Availability vocabulary and strategy | Availability vocabulary, Multi-AZ by default, The four DR strategies, Active-active vs active-passive |
| Scaling and caching | Auto Scaling everywhere, Elastic Load Balancing for resilience, Caching for performance and protection |
| Data protection and failover | AWS Backup in depth, Database failover across Regions, Failover with Route 53, Multi-AZ, replicas, backups and PITR, ElastiCache (Valkey/Redis OSS) |
| Resilience in code | Resilience patterns in .NET, Chaos engineering with FIS, Resilience Hub and game days |
| Monitoring | Module 11 overview, Alarms and composite alarms, Dashboards and Synthetics canaries, Lab: observe a banking API |
| The lab itself | Lab: Resilient Account API |
What is in the repository
| Where | What |
|---|---|
labs/m12-resilient-account-api | The resilience library (Polly pipeline, load shedding, readiness gate, AZ helpers, capacity and DR calculators), CDK and Terraform for a multi-AZ Fargate service, the DR plan template, 86 tests |
labs/m02-secure-vpc, labs/m11-observe-banking-api | The isolation pattern and the observability patterns, referenced |
projects/p5-ha-banking-app | The capstone: README.md, checklist.md, docs/ (architecture, five ADRs, security controls, testing, monitoring, cost and teardown, DR drill guide), infra/terraform (network, data, compute, backup and DR, monitoring), the drill tool src/Harbourline.HaBanking.DrDrill with 18 tests, and drills/sample-drill.json |
Build path
Run the tests (free). The m12 lab (86 tests) and the drill tool (18 tests) in Docker. Run the drill tool on the sample log and read the report it prints.
Do the tabletop drill (free). Walk the runbook in the DR plan template with the sample numbers, and write down where you think the time goes before you measure anything.
Deploy the stack. terraform apply in infra/terraform with your owner name; wait about 20 to 30 minutes for the Multi-AZ database. Open the ALB address and confirm three healthy targets across three AZs.
Break things in order. Stop a task, scale under load, restore a backup, fail the database over, fail the cache over, simulate an AZ loss. Record the result and time of each.
Add the DR layer and run the drill. Set enable_dr_replica = true, apply, run the game day from the DR drill guide, save the event log and produce the report.
Tear down both Regions, delete recovery points first, and prove nothing is left.
What was verified when this page was written
| Check | Result |
|---|---|
| Lab m12 suite (Docker, .NET 10 SDK) | 86 passed, 0 failed |
| Drill tool suite | 18 passed, 0 failed; the sample drill reports Verdict Pass, RTO about 24 minutes, RPO 48 seconds |
terraform validate on infra/terraform (Terraform 1.14, AWS provider 6.x), including the DR-Region provider alias | Valid |
| Not run | terraform apply, any AWS deployment, a restore, a failover or a real drill. Prices are estimates; engine and Valkey versions, CLI flags and the FIS and zonal-shift steps are marked ⚠ VERIFY in the repository docs |
Security controls
The repository's docs/security-controls.md lists eleven controls with how to verify each. The ones that define this design:
| Control | How it is shown |
|---|---|
| Data tier is isolated | The data route table has only the local route; Reachability Analyzer from the internet gateway to the database interface returns not reachable |
| Each hop is allowed by reference | ALB to tasks to database and cache by security-group ID, never CIDR; the default security group allows nothing |
| Encryption everywhere | Customer-managed KMS key with rotation for the database, cache, logs and alarm topic; a separate key in the DR Region; TLS in transit |
| No stored passwords | RDS creates and rotates the master secret; the task execution role can read only that secret |
| Backups survive their source | Cross-Region copy with its own key (production adds a separate account and Vault Lock) |
| Safe deployments | ECS circuit breaker with rollback; deletion protection variable |
Honest gaps: no WAF and an open ALB in the lab, one NAT gateway, backups in the same account, an empty task role because no real application is wired in, and no GuardDuty or Config here (the landing-zone capstone enables them organisation-wide).
Cost and teardown
Roughly USD 0.20 per hour while deployed (NAT gateway, ALB, three small Fargate tasks, a Multi-AZ database, two cache nodes, alarms and logs), about USD 145 a month if forgotten. A six-hour session is under USD 1.50. The optional DR replica adds about USD 0.02 per hour plus data transfer. Prices are list-price estimates dated 2026-11-18, not re-fetched from the pricing pages ⚠ VERIFY.
Teardown checklist
Delete the recovery points in both vaults (Terraform cannot destroy a vault that holds them), run terraform destroy, delete any manual snapshots and restored test instances, then run the tag query for Project=cea-lab and Lab=p5-ha-banking-app in both Regions. KMS keys sit in a 7-day pending-deletion state. The repository's docs/cost-and-teardown.md has the ordered commands.
Acceptance criteria
- The m12 and drill-tool tests pass; the sample drill reports Pass.
- Three targets are healthy across three AZs, and the data tier has no route to the internet.
- You stopped a task, failed the database over and failed the cache over, with times recorded and no customer-visible errors beyond a short blip.
- A backup was restored and the data checked; the restored instance was deleted.
- The customer-impact alarm did not fire during the simulated AZ loss, and did fire when you broke the application.
- A drill report exists with measured RTO and RPO and an action for every miss.
- Both Regions are destroyed and nothing tagged for the project remains.
Portfolio-ready checklist
The full list is projects/p5-ha-banking-app/checklist.md. The essentials:
- Five ADRs: three-tier three-AZ VPC, Multi-AZ database and cache, Fargate with target tracking, pilot light with cross-Region backup, page on symptoms.
- Diagrams that match what you deployed, including the failover sequence.
- Evidence you collected: the isolation proof, the failure-test table with times, the restore time, the drill report.
- A written list of what production would still need: WAF, HTTPS, a NAT gateway per AZ, a separate backup account with Vault Lock, an agreed impact tolerance, a regulator-ready test schedule.
- Extensions: wire in the Module 3 API with Npgsql and a cache-aside layer, a Synthetics canary, an alarm forwarder from the DR Region, Aurora Global Database as the managed cross-Region option, AWS Fault Injection Service experiments.
Exercises
🟢 Guided. Change drills/sample-drill.json so that the database promotion takes 25 minutes. Predict the verdict and the slowest step before you run the tool, then check.
🟡 Challenge. Add a HttpCode_ELB_5XX_Count alarm to the customer-impact composite alarm, and explain in one sentence the difference between an ELB 5xx and a target 5xx for a customer's experience.
🔴 Live problem. Ticket INC-2043: "After last night's Multi-AZ failover the Account API threw errors for eleven minutes although the database was back after ninety seconds. The alarms said everything was fine." Name three places the eleven minutes could have gone (think about connection pools, DNS caching and health checks) and what evidence you would collect for each.
Interview questions
- Your database is Multi-AZ. Does it need backups? Why?
- How would you prove your RTO to an auditor, and why is "the database was promoted" the wrong end of the stopwatch?
- What happens to in-flight requests during an RDS failover, and what must the application do about it?
- Why is the minimum task count tied to the number of Availability Zones?
- When is a pilot-light DR strategy the wrong choice?
Key takeaways
- Availability is designed in layers: tasks, AZs, backups, Region. Each layer fails differently.
- A standby is not a backup, and a backup is not a failover plan.
- An RTO you have not timed is a guess; measure it with a drill and write down the misses.
- Page on what customers feel; ticket on causes.
- Always count the cost of the safety: NAT, Multi-AZ and a second Region are the bill.
Related
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Capstone 4: Complete .NET CI/CD
A delivery pipeline an auditor can accept as a control: GitHub Actions with OIDC and no stored keys, one immutable image promoted by digest, security scanning gates, an independent production approval, ECS-native blue/green with alarm rollback, and an append-only evidence record, built on the Module 9 lab plus Terraform for the environments.
Capstone 6: Legacy .NET Framework migration
Move the core of a fictional legacy .NET Framework estate to AWS in six stages: assess with the 7 Rs, port and containerise (a plan, since the legacy code does not exist), migrate SQL Server to RDS PostgreSQL with DMS, prove the data matches with a tested reconciler, cut over behind a go/no-go gate with weighted DNS, and keep a rollback that has been rehearsed. Built on the Module 13 lab plus Terraform and a small tested cut-over tool.