Multi-AZ by default: RDS, Aurora, ElastiCache, ECS and your own services
Know exactly what Multi-AZ means for each service Harbourline uses, which parts AWS does for you and which parts are your job, and make your own stateless .NET services AZ-aware and failover-tolerant.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · EF Core 10 with Npgsql · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- State, for RDS, Aurora, ElastiCache, DynamoDB, S3, ECS and Lambda, whether Multi-AZ is automatic, optional or your responsibility.
- Choose between an RDS Multi-AZ DB instance, an RDS Multi-AZ DB cluster and Aurora for a banking workload, with failover behaviour as one of the criteria.
- Spot the zonal resources that quietly break a "Multi-AZ" design (NAT gateway, single-AZ subnets, one-replica clusters, shared state on one task).
- Make a .NET service failover-tolerant: retry on transient errors, reconnect to the new primary, and carry the AZ on logs and metrics.
- Check that tasks really are spread across AZs with a small, tested C# helper.
- Trigger and time a failover safely.
Prerequisites
- Availability vocabulary, Auto Scaling everywhere and ELB for resilience
- Multi-AZ, replicas, backups and PITR and Amazon Aurora — the database mechanics. This lesson summarises them for decision-making and adds the application side.
- ElastiCache and Valkey
- Tools: Docker for the lab tests; optional AWS CLI v2 with
learn-sandbox
Why this matters in banking
Harbourline's architecture review says "everything is Multi-AZ". An engineer then tests it by disabling one AZ's networking in a pre-production account, and three things break:
- The one NAT gateway was in the failed AZ, so every task in the other two AZs lost outbound internet access and could not reach a third-party sanctions API.
- The Aurora cluster had one writer and no reader. Failover needed a brand-new instance to be created, taking minutes instead of seconds.
- A background job ran as a singleton on one task; it happened to be on the failed AZ and silently stopped for forty minutes.
"Multi-AZ" is a property you verify per component, not a label you apply to an architecture. This lesson gives you the checklist.
The idea in plain language
Multi-AZ means a service keeps working if one Availability Zone fails, because the service has capacity or copies of data in at least one other AZ. There are three flavours, and it matters which one you are getting:
| Flavour | What it means | Examples |
|---|---|---|
| Built in, regional by design | The service spans AZs and you cannot choose otherwise | DynamoDB, S3 Standard, SQS, SNS, EventBridge, Lambda, Step Functions, API Gateway |
| Optional, you switch it on | Standby or replicas exist in another AZ only if you configure them | RDS Multi-AZ, ElastiCache Multi-AZ, EFS regional vs One Zone, Aurora readers |
| Your design | You must place and balance the capacity | EC2 and ECS capacity, NAT gateways, your own caches, your background jobs |
Analogy: the bank keeps its records in a vault. Some vaults come with a duplicate vault in another building automatically (built in). Some offer one if you pay and ask (optional). And your own staff rota across buildings is a thing you design (your design). A review that says "we are in two buildings" must check each of the three.
What each service does
Amazon RDS (SQL Server, PostgreSQL, MySQL and others) has two Multi-AZ shapes:
- Multi-AZ DB instance deployment: one primary and one standby in another AZ. The standby is not readable. Replication to it is synchronous at the storage or engine level (the mechanism varies by engine). On failure, RDS flips the DNS name of the instance endpoint to the standby. Failover typically takes 60 to 120 seconds, as the earlier databases lesson explains; measure yours.
- Multi-AZ DB cluster deployment: one writer and two readable standbys in three AZs, using the engine's native semi-synchronous replication (a commit needs acknowledgement from at least one reader). The readers serve read traffic and are failover targets. Failover first resolves replica lag, then promotes, so lag affects failover time. The documentation lists supported engines, instance classes and Regions, which are limited ⚠ VERIFY before choosing this for SQL Server. It is not the same thing as Aurora.
Amazon Aurora separates compute from storage. The cluster volume keeps six copies of the data across three AZs, and writes are replicated across AZs synchronously, regardless of how many DB instances you run. That protects the data even with a single instance. For availability of the instances:
- If the cluster has one or more Aurora Replicas, a replica is promoted on failure. AWS documents service as typically restored in less than 60 seconds, often less than 30.
- If the cluster has no replicas, Aurora recovers the primary host in the same AZ, which AWS documents as typically less than 10 minutes. If the AZ itself is down, you must create an instance in another AZ yourself.
- You can set promotion tiers (0 highest priority to 15) to choose which replica is promoted.
- Connect to the cluster endpoint (always the current writer) and the reader endpoint (load-balances readers). RDS Proxy can bypass DNS caching to shorten failover, and preserves connections.
Amazon ElastiCache (Valkey, Redis OSS) with Multi-AZ enabled spreads replicas across AZs and enables automatic failover: on primary failure the replica with the least replication lag is promoted and the primary endpoint's DNS is updated. Writes typically resume within seconds. Replication is asynchronous, so a small amount of data can be lost, which is fine for a cache and a reason not to treat a cache as a system of record. You need at least one replica in a different AZ. A customer-initiated reboot of a primary does not trigger failover.
DynamoDB, S3, SQS, SNS, EventBridge: replicated across multiple AZs by the service. There is no setting to enable. S3 One Zone storage classes and EFS One Zone are the exceptions where you choose a single AZ.
ECS and Fargate: you give the service subnets in multiple AZs, and ECS spreads tasks across them (with AZ rebalancing available). EC2 launch type capacity spreads via the Auto Scaling group's subnets. Spread only works if each AZ has subnets, capacity and quota.
Lambda: runs across AZs automatically. If the function is attached to a VPC, give it subnets in at least two AZs.
Zonal resources that you place yourself:
| Resource | Zonal? | What to do |
|---|---|---|
| NAT gateway | Yes: lives in one subnet, one AZ | One per AZ, with each AZ's route table pointing to its own |
| Interface VPC endpoints | One network interface per AZ you select | Select every AZ you use |
| EC2 instances, EBS volumes | Yes | Spread instances; EBS is tied to one AZ, so use snapshots or replicate at the application layer |
| Subnets | Exactly one AZ each | Create one per tier per AZ |
| Load balancer nodes | One per enabled AZ | Enable all AZs you use; no empty subnets |
Your own .NET services: the stateless rules
A service survives an AZ loss only if any copy can serve any request:
- No local state that matters. Sessions, rate-limit counters, idempotency records and caches that must be shared belong in DynamoDB, ElastiCache or the database.
- Share ASP.NET Core Data Protection keys. Otherwise a cookie or antiforgery token issued by one task cannot be read by another.
- Singleton background work needs coordination. If a hosted service must run once, use a lease (conditional write in DynamoDB), a queue consumer, or a scheduled service such as EventBridge Scheduler. "One task runs it" is a single point of failure.
- Idempotent operations. After a failover, a client or the platform may retry. Money movements need idempotency keys.
- Reconnect gracefully. Connection pools hold connections to the old primary. They break and must be re-established.
- Know your AZ. Put the AZ on every log line and metric dimension; during an incident, "all errors come from one AZ" is the fastest diagnosis.
What problem the AWS service solves
Multi-AZ features convert infrastructure failure (a data centre problem) into a brief blip instead of an outage. They do it with no change to your data model, which is why they are the cheapest resilience you can buy.
Honest limits:
- Failover is not instant. Seconds to minutes, during which writes fail. Your client code must tolerate that.
- Standbys cost money (a Multi-AZ DB instance costs about double a single-AZ instance ⚠ VERIFY current pricing). Aurora replicas are full instances.
- It protects against infrastructure loss, not bad data. A bad
UPDATEreplicates. Use point-in-time recovery. - It does not protect against a Region failure. That is DR (The four DR strategies).
- Failover must be tested. A Multi-AZ configuration that has never failed over is an assumption.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| SQL Server Always On availability group (synchronous replica) | RDS Multi-AZ DB instance | Managed; standby not readable; failover by DNS flip; for SQL Server the mechanism depends on edition and engine ⚠ VERIFY |
| Azure SQL zone-redundant configuration | RDS Multi-AZ, or Aurora | Aurora's storage is replicated across AZs even with one instance |
| Read-scale Always On secondary | Multi-AZ DB cluster readers, Aurora Replicas | Aurora: up to 15 replicas, endpoint load-balances |
| Azure Cache for Redis zone redundancy | ElastiCache Multi-AZ with automatic failover | Replicas must exist in other AZs |
| Zone-redundant App Service / AKS node pools in zones | ECS service across subnets in AZs; ASG across subnets | You choose subnets per AZ; NAT gateways are zonal, whereas Azure NAT Gateway is zonal too, so the same trap applies |
| Cosmos DB with zone redundancy | DynamoDB (always multi-AZ) | Nothing to enable |
SQL connection resiliency (EnableRetryOnFailure) | EF Core with Npgsql EnableRetryOnFailure | Same EF Core feature; use it with the cluster or proxy endpoint |
Architecture
What happens when AZ a fails:
Security-boundary view: standbys and replicas must have the same controls as the primary. Encryption must be on (a KMS key available in the AZ's Region, which is the same Region), security groups identical, and logging enabled. An unencrypted replica or a differently-permissioned standby is a quiet compliance gap.
Step-by-step implementation
Inventory every component and its AZ behaviour
Make a table: component, regional/zonal/optional, current AZ count, what happens on AZ loss, how it was proven. Include the unglamorous ones (NAT gateways, VPC endpoints, schedulers, secrets rotation Lambda, cron-style tasks).
Choose the database shape
For PostgreSQL-compatible OLTP with the shortest failover, choose Aurora with at least one reader in another AZ. For SQL Server (which Harbourline's legacy estate uses), choose RDS Multi-AZ and plan for the 60 to 120 second failover. Use a Multi-AZ DB cluster where the engine supports it and you want readable standbys without Aurora.
Put capacity in every AZ
Three subnets per tier, tasks spread across them, NAT per AZ, endpoints in each AZ. Enable ECS service AZ rebalancing. Use the AZ-loss capacity formula from the scaling lesson.
Make the application failover-tolerant
Use the cluster endpoint (never an instance endpoint), set connection and command timeouts, enable EF Core EnableRetryOnFailure, keep operations idempotent, and consider RDS Proxy to shorten failover and absorb connection storms.
Add AZ awareness to telemetry
Read the AZ from the ECS task metadata endpoint at start-up and add it to log scope and metric dimensions. Verify spread with AzSpreadChecker.
Fail it on purpose
In a non-production account, force a database failover and watch what your service does. Measure write outage seconds. If you never measured it, you do not know it.
AWS Console instructions
- RDS Multi-AZ instance: Console → RDS → Databases → your database → Modify → Availability and durability → Multi-AZ deployment. Force a failover with Actions → Reboot → Reboot with failover.
- Aurora: Console → RDS → Databases → your cluster → Add reader (choose a different AZ). Force failover with Actions → Failover.
- ElastiCache: Console → ElastiCache → Valkey caches (or Redis OSS) → your cache → Modify → Multi-AZ; test with the failover action on the node/shard page.
- See task AZ spread: Console → ECS → Clusters → your cluster → Services → your service → Tasks; the task detail shows the Availability Zone.
Console labels move; search the page for "Multi-AZ" or "failover" if a label differs ⚠ VERIFY.
AWS CLI
PROFILE=learn-sandbox
REGION=us-east-1
# Which AZ is each Aurora instance in, and which one is the writer?
aws rds describe-db-clusters --db-cluster-identifier hl-m12-demo --profile $PROFILE --region $REGION \
--query "DBClusters[0].DBClusterMembers[].{Instance:DBInstanceIdentifier,Writer:IsClusterWriter,Tier:PromotionTier}" --output table
aws rds describe-db-instances --profile $PROFILE --region $REGION \
--query "DBInstances[].{Id:DBInstanceIdentifier,AZ:AvailabilityZone,MultiAZ:MultiAZ,Engine:Engine}" --output table
# Force an RDS Multi-AZ failover (reboot with failover) and an Aurora failover. Non-production only.
aws rds reboot-db-instance --db-instance-identifier hl-m12-demo-sql --force-failover --profile $PROFILE --region $REGION
aws rds failover-db-cluster --db-cluster-identifier hl-m12-demo --profile $PROFILE --region $REGION
# ElastiCache: confirm Multi-AZ and automatic failover, then test a failover (limited per 24 hours).
aws elasticache describe-replication-groups --replication-group-id hl-m12-cache --profile $PROFILE --region $REGION \
--query "ReplicationGroups[0].{MultiAZ:MultiAZ,AutoFailover:AutomaticFailover,Status:Status}"
aws elasticache test-failover --replication-group-id hl-m12-cache --node-group-id 0001 --profile $PROFILE --region $REGION
# Count NAT gateways per AZ (you want one per AZ you use) and look for single-AZ interface endpoints.
aws ec2 describe-nat-gateways --filter Name=state,Values=available --profile $PROFILE --region $REGION \
--query "NatGateways[].{Id:NatGatewayId,Subnet:SubnetId}" --output table
aws ec2 describe-vpc-endpoints --profile $PROFILE --region $REGION \
--query "VpcEndpoints[?VpcEndpointType=='Interface'].{Svc:ServiceName,Subnets:SubnetIds}" --output jsonThe identifiers above are examples; the lab stack does not create a database, so use the Module 6 lab database or a small database you create and delete in the same hour.
Infrastructure as Code
The lab stack applies the Multi-AZ rules that do not need a database: three AZs, tasks spread across them, a load balancer in all three and ECS AZ rebalancing. For the database part, the pattern is below.
// Aurora PostgreSQL with a writer plus a reader in another AZ (illustrative excerpt).
var cluster = new DatabaseCluster(this, "Aurora", new DatabaseClusterProps
{
Engine = DatabaseClusterEngine.AuroraPostgres(new AuroraPostgresClusterEngineProps
{
Version = AuroraPostgresEngineVersion.VER_16_6, // use a version your Region supports ⚠ VERIFY
}),
Vpc = vpc,
VpcSubnets = new SubnetSelection { SubnetType = SubnetType.PRIVATE_ISOLATED },
Writer = ClusterInstance.Provisioned("writer", new ProvisionedClusterInstanceProps
{
InstanceType = InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE),
PromotionTier = 0,
}),
Readers =
[
ClusterInstance.Provisioned("reader1", new ProvisionedClusterInstanceProps
{
InstanceType = InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE),
PromotionTier = 0, // first choice for promotion
}),
],
StorageEncrypted = true,
DeletionProtection = true,
BackupRetention = Duration.Days(14),
});
// CDK spreads the instances across the AZs of the chosen subnets. Confirm with describe-db-instances.These database excerpts are illustrative and are not part of the lab stack, to keep the lab cheap ⚠ VERIFY argument names against your CDK and provider versions before using them.
Complete code
Two helpers from the lab, EcsAvailabilityZoneProvider and AzSpreadChecker (both covered by AzAwarenessTests), and how a service uses them with EF Core.
public sealed class EcsAvailabilityZoneProvider(HttpClient http, string? metadataUri)
{
public const string Unknown = "unknown";
public async Task<string> GetAvailabilityZoneAsync(CancellationToken cancellationToken = default)
{
if (string.IsNullOrWhiteSpace(metadataUri))
{
return Unknown; // not on ECS (a laptop, a test)
}
try
{
using var response = await http.GetAsync($"{metadataUri.TrimEnd('/')}/task", cancellationToken);
response.EnsureSuccessStatusCode();
await using var body = await response.Content.ReadAsStreamAsync(cancellationToken);
using var document = await JsonDocument.ParseAsync(body, cancellationToken: cancellationToken);
return document.RootElement.TryGetProperty("AvailabilityZone", out var az) && az.GetString() is { Length: > 0 } name
? name
: Unknown;
}
catch (HttpRequestException) { return Unknown; } // diagnostics must never take the service down
catch (JsonException) { return Unknown; }
}
}
public static class AzSpreadChecker
{
public sealed record Result(IReadOnlyDictionary<string, int> TasksPerAz, int Skew, bool MeetsMinimumAzs, bool IsBalanced);
public static Result Evaluate(IEnumerable<string> azPerTask, int expectedAzCount)
{
var counts = azPerTask.GroupBy(az => az).ToDictionary(g => g.Key, g => g.Count());
var populated = counts.Count;
var highest = counts.Count == 0 ? 0 : counts.Values.Max();
var lowest = populated < expectedAzCount || counts.Count == 0 ? 0 : counts.Values.Min();
var skew = highest - lowest;
return new Result(counts, skew, populated >= expectedAzCount, skew <= 1);
}
}using Harbourline.Resilience;
using Microsoft.EntityFrameworkCore;
var builder = WebApplication.CreateBuilder(args);
// The cluster endpoint always points at the current writer; never use an instance endpoint.
var connectionString = builder.Configuration.GetConnectionString("Accounts")
?? throw new InvalidOperationException("ConnectionStrings:Accounts is required.");
builder.Services.AddDbContext<AccountsDbContext>(options =>
options.UseNpgsql(connectionString, npgsql =>
{
// Retry transient errors, including the connection errors seen during a failover.
npgsql.EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(10), errorCodesToAdd: null);
npgsql.CommandTimeout(15);
}));
// Find out which AZ this task runs in, once, and add it to every log line.
var azProvider = new EcsAvailabilityZoneProvider(
new HttpClient { Timeout = TimeSpan.FromSeconds(2) },
Environment.GetEnvironmentVariable("ECS_CONTAINER_METADATA_URI_V4"));
var availabilityZone = await azProvider.GetAvailabilityZoneAsync();
var app = builder.Build();
app.Use(async (context, next) =>
{
using (app.Logger.BeginScope(new Dictionary<string, object> { ["az"] = availabilityZone }))
{
await next(context);
}
});
app.MapGet("/accounts/{id}", async (string id, AccountsDbContext db, CancellationToken ct) =>
await db.Accounts.FindAsync([id], ct) is { } account ? Results.Ok(account) : Results.NotFound());
app.Run();
public sealed class AccountsDbContext(DbContextOptions<AccountsDbContext> options) : DbContext(options)
{
public DbSet<Account> Accounts => Set<Account>();
}
public sealed record Account(string AccountId, string Currency, decimal Balance);The second excerpt is illustrative and needs the Npgsql.EntityFrameworkCore.PostgreSQL package and a mapped Account key; the lab tests cover the helpers, not the web host.
Code explained line by line
ECS_CONTAINER_METADATA_URI_V4is an environment variable that ECS sets on each container (Fargate platform version 1.4.0 and later). Its/taskpath returns JSON that includes anAvailabilityZoneproperty.EcsAvailabilityZoneProviderreturnsunknownfor anything abnormal (not on ECS, endpoint error, malformed JSON): a diagnostics helper must never be the reason a service fails to start.AzSpreadChecker.Evaluatecounts tasks per AZ. An expected AZ with no tasks counts as zero, so "all tasks in one AZ" has a skew equal to the task count, and a one-task difference between AZs is accepted as balanced.EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: 10 s)turns EF Core's execution strategy on, retrying transient failures with increasing delay. Together with a cluster endpoint, this rides out a typical failover. A non-idempotent write inside a retry can run twice if the first attempt committed but the response was lost, so use transactions and idempotency keys for money movements.CommandTimeout(15)bounds one command, so a hung connection to the old writer does not pin a request for minutes.- The
app.Usemiddleware putsazinto the logging scope, so every log line from a request carries it.
Security considerations
- Encrypt every replica and standby with KMS, and keep the same security groups and IAM database authentication posture. Aurora and RDS Multi-AZ inherit encryption from the primary, but verify.
- Secrets across failover. Use Secrets Manager with managed master credentials, and make sure the application reads the secret at connection time (or refreshes), so a rotation or failover does not strand it.
- RDS Proxy adds an IAM and secret-based authentication path; scope its role to the single secret.
- Do not expose instance endpoints. They bypass failover and tempt people to hard-code a host.
- Failover testing is a change. Do it through a controlled, ticketed process, with CloudTrail evidence, never in production without approval.
⚠ Learning shortcut — not for production
The lab and these excerpts use small instance classes, a single secret and DeletionProtection shown for illustration. They tolerate an AZ failure but are not a performance or compliance design. In production, size from load tests, add RDS Proxy where connection counts are high, turn on audit logging, set backup retention to your policy, and have compliance and the data-protection officer review where the data may live (UK GDPR) before choosing Regions.
Banking use case
Harbourline's Accounts API reads and writes an Aurora PostgreSQL cluster, caches balances in ElastiCache for Valkey and sends account events to SQS. The Multi-AZ review table, abbreviated:
| Component | AZ behaviour | Evidence |
|---|---|---|
| ALB and ECS tasks | 3 AZs, 18 tasks (6 per AZ), AZ rebalancing on | Spread check in CI, drill |
| Aurora | Writer plus two readers across 3 AZs, promotion tier 0 on one reader | Failover drill: 24 s measured outage |
| Valkey | Primary plus two replicas in 3 AZs, Multi-AZ on | test-failover event log |
| SQS, DynamoDB idempotency table | Regional by design | AWS documentation |
| NAT gateways | One per AZ | IaC test |
| Nightly interest accrual job | Scheduled by EventBridge Scheduler, queue-driven workers | Not tied to one task |
The numbers are fictional teaching values.
Common mistakes
- One NAT gateway for the VPC. An AZ failure takes outbound access away from other AZs.
- Aurora with no reader and the belief that "Aurora is Multi-AZ". The storage is; the compute may not be.
- Using instance endpoints. They do not follow failover.
- Treating a cache as a database. Replication to the replica is asynchronous.
- Retrying non-idempotent writes without idempotency keys.
- Long DNS caching or connection pools that never recycle. The app keeps talking to the old writer. Set sensible pool lifetimes and use the cluster endpoint or RDS Proxy.
- Singleton jobs. "Runs on task 1" is not a design.
- Never testing failover. The first test is then an incident.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Writes fail for minutes after Aurora failover | No reader to promote, or app caching the old IP/DNS | Add a reader in another AZ; use the cluster endpoint; set pool lifetime; consider RDS Proxy |
After failover the app throws read-only transaction errors | Connected through an old connection to a demoted node | Retry; recycle connections; use cluster endpoint |
| RDS Multi-AZ failover takes longer than expected | Large crash recovery, long-running transactions | Measure, keep transactions short, consider Aurora |
| Tasks all in one AZ | Single subnet, capacity shortage, rebalancing off | Add subnets; enable rebalancing; check quotas |
| Outbound calls fail from two AZs when one AZ has an incident | Single NAT gateway | One NAT per AZ with per-AZ route tables |
ElastiCache test-failover fails or is rejected | Multi-AZ/automatic failover not enabled, or limit of 15 shard tests per 24 hours | Enable Multi-AZ; stay within the limit |
AzSpreadChecker shows skew of 2 or more | Rebalancing lag or an AZ shortage | Check ECS service events and subnet IP space |
Cost and cleanup
Reading the lesson and running the helpers locally costs nothing. If you experiment with real databases, an Aurora cluster with a writer and reader costs several dollars per day even at small sizes ⚠ VERIFY current prices; an RDS Multi-AZ instance costs about twice a single-AZ instance; ElastiCache nodes bill hourly. Create, test failover, and delete within an hour, and take no final snapshot unless you want one.
# Delete a lab Aurora cluster (instances first, then cluster) with no final snapshot. Non-production only.
aws rds delete-db-instance --db-instance-identifier hl-m12-demo-0 --skip-final-snapshot --profile learn-sandbox --region us-east-1
aws rds delete-db-instance --db-instance-identifier hl-m12-demo-1 --skip-final-snapshot --profile learn-sandbox --region us-east-1
aws rds delete-db-cluster --db-cluster-identifier hl-m12-demo --skip-final-snapshot --profile learn-sandbox --region us-east-1
# Confirm nothing tagged for this lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
--profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"If deletion protection is on, disable it first (modify-db-cluster --no-deletion-protection).
Exercises
- 🟢 Guided: Run
AzAwarenessTests. Then write a table of your own account's resources (or the Harbourline estate in this lesson) with columns regional, optional, zonal, and mark every zonal one. - 🟡 Challenge: An architecture has: 3-AZ ECS service, one NAT gateway, an RDS Multi-AZ SQL Server instance, an ElastiCache Valkey cluster with one node, a nightly job on one task. List every component that fails the "survives an AZ loss" test and give the fix and its approximate cost direction.
- 🔴 Live problem: "We forced an Aurora failover in staging and the API returned 500s for 4 minutes, but AWS says under a minute." Investigate: what do you check first, second and third, and what changes do you propose?
Assignment
Write docs/multi-az-review.md for Harbourline's Payments service: a component table (regional, optional, zonal), the AZ-loss behaviour of each, the evidence you would collect, and a failover drill plan with the success criteria (maximum write outage seconds, zero lost acknowledged payments).
Solutions
Regional by design: DynamoDB, S3 Standard, SQS, SNS, Lambda, API Gateway. Optional: RDS Multi-AZ, Aurora replicas, ElastiCache Multi-AZ, EFS regional. Zonal (your responsibility): NAT gateways, EC2/EBS, subnets, interface endpoint ENIs, load balancer nodes. The point is that the zonal list contains things people forget, and each needs a per-AZ copy or an explicit decision.
Fail: (1) one NAT gateway: add one per AZ and per-AZ route tables (cost: two more NATs and their data processing); (2) the one-node Valkey cluster: add replicas in other AZs and enable Multi-AZ (cost: replica nodes); (3) the nightly job on one task: move to EventBridge Scheduler plus a queue and worker service, or a DynamoDB lease (low cost). The RDS Multi-AZ SQL Server instance passes the AZ test but has a 60 to 120 second failover, so confirm that fits the RTO, and the ECS service passes if capacity per AZ holds peak after a loss.
First check whether the cluster had a reader in another AZ at the time (if not, it created a new writer, which takes minutes). Second, check how the app connects: instance endpoint instead of cluster endpoint, long DNS TTL or connection pool holding broken connections, no retry policy. Third, check the failover event timeline in RDS events versus your application's error timeline to see whether the delay was the database or the client. Changes: add a reader with promotion tier 0 in another AZ, use the cluster endpoint, enable retry-on-failure and reasonable pool lifetimes, consider RDS Proxy, and re-measure with the failover stopwatch from the databases lesson. Record the new measured number in the runbook.
Knowledge check
Answer: The data is: Aurora keeps six copies across three AZs regardless of instance count. The service is not necessarily: with no reader, failover means recovering the host or creating a new instance, which takes longer, and if the whole AZ is down you must create an instance in another AZ. Add a reader in a different AZ.
Answer: A NAT gateway lives in one subnet in one AZ. If that AZ fails, private subnets in other AZs that route through it lose outbound connectivity. Deploy one per AZ and point each AZ's route table at its own.
Answer: The instance deployment has one non-readable standby using synchronous replication and DNS failover. The cluster deployment has a writer and two readable standbys in three AZs using semi-synchronous replication, with the readers serving read traffic. Multi-AZ DB clusters are not Aurora and are available for specific engines, instance classes and Regions.
Answer: The cluster endpoint always points at the current writer, so it follows failover. An instance endpoint points to one specific instance, which may be demoted or down.
Answer: A small amount of recent writes, because replication to replicas is asynchronous and the promoted replica (the one with the least lag) may not have received the latest changes. Treat the cache as disposable, not a system of record.
Interview questions
- "Is Aurora Multi-AZ by default?" Its storage is replicated across three AZs by default. Instance-level availability depends on having at least one Aurora Replica in another AZ to promote; otherwise recovery is slower.
- "How would you prove a service survives an AZ loss?" Inventory zonal components, ensure capacity per AZ meets the AZ-loss formula, then run a game day that removes an AZ (for example with FIS or a zonal shift in pre-production) and measure errors and recovery time.
- "What is the most common hidden single point of failure in a Multi-AZ VPC?" A single NAT gateway, followed by singleton background jobs and single-replica caches.
- "How do you make an EF Core app handle failover?" Cluster or proxy endpoint, command timeouts,
EnableRetryOnFailure, idempotent writes, pool lifetime tuning, and logs that carry the AZ.
Certification practice
A company runs a relational database that must remain available if an Availability Zone fails, with automatic failover, and the standby capacity should also serve read queries to reduce load on the primary. The engine is PostgreSQL. Which option fits?
A. RDS Multi-AZ DB instance deployment. B. RDS Multi-AZ DB cluster deployment (or Aurora with replicas). C. RDS Single-AZ with a nightly snapshot. D. A cross-Region read replica.
Answer: B. Readable standbys in other AZs with automatic failover come from a Multi-AZ DB cluster or Aurora replicas. A has a non-readable standby. C offers no automatic failover. D is for Regional DR and is asynchronous with manual promotion.
An Aurora cluster with one instance in a single Availability Zone is being prepared for production. The operations team wants the fastest automated recovery from an AZ failure. What should they do?
A. Increase the backup retention. B. Add an Aurora Replica in a different Availability Zone. C. Enable enhanced monitoring. D. Move to a larger instance class.
Answer: B. A replica in another AZ can be promoted, typically within a minute. A, C and D do not provide a failover target.
A VPC has private subnets in three Availability Zones and one NAT gateway in a public subnet in AZ a. When AZ a fails, instances in AZs b and c cannot reach the internet. What is the most resilient fix?
A. Add a larger NAT gateway. B. Create a NAT gateway in each Availability Zone and route each private subnet to the NAT gateway in its own AZ. C. Attach an Elastic IP to each instance. D. Move all instances to AZ a.
Answer: B. NAT gateways are zonal. One per AZ, with per-AZ routing, removes the cross-AZ dependency. A keeps the single AZ. C is a poor, insecure substitute. D reduces resilience.
Completion checklist
- I can say whether Multi-AZ is built in, optional or my responsibility for each service I use
- I can explain the difference between RDS Multi-AZ instance, Multi-AZ DB cluster and Aurora
- I ran
AzAwarenessTestsand understood the spread checker - I wrote a Multi-AZ review for one service
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- Multi-AZ is built in for some services, optional for others, and your design for compute, NAT and jobs. Verify each.
- Aurora protects data across three AZs by default, but you need a reader in another AZ for fast instance failover.
- Failover takes seconds to minutes; your application must retry and reconnect, and money movements must be idempotent.
- NAT gateways, singleton jobs and single-node caches are the classic hidden single points of failure.
- Measure failover. An untested Multi-AZ design is an assumption.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Elastic Load Balancing for resilience: cross-zone, health checks and zonal shift
Use load balancer behaviour as a resilience tool: choose cross-zone settings, design health checks that detect real failure without causing outages, drain targets cleanly, and move traffic away from a sick Availability Zone.
Resilience patterns in .NET: timeouts, retries, breakers, bulkheads and load shedding
Make a .NET API survive a slow or failing dependency with a layered resilience pipeline, a retry budget, a circuit breaker, a bulkhead, load shedding and a graceful fallback, and prove each layer with a fault-injecting test backend.