Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Multi-AZ by default: RDS, Aurora, ElastiCache, ECS and your own services

Know exactly what Multi-AZ means for each service Harbourline uses, which parts AWS does for you and which parts are your job, and make your own stateless .NET services AZ-aware and failover-tolerant.

intermediate⏱ 1 h 25 min🟢 FreeSAA-C03SOA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · EF Core 10 with Npgsql · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • State, for RDS, Aurora, ElastiCache, DynamoDB, S3, ECS and Lambda, whether Multi-AZ is automatic, optional or your responsibility.
  • Choose between an RDS Multi-AZ DB instance, an RDS Multi-AZ DB cluster and Aurora for a banking workload, with failover behaviour as one of the criteria.
  • Spot the zonal resources that quietly break a "Multi-AZ" design (NAT gateway, single-AZ subnets, one-replica clusters, shared state on one task).
  • Make a .NET service failover-tolerant: retry on transient errors, reconnect to the new primary, and carry the AZ on logs and metrics.
  • Check that tasks really are spread across AZs with a small, tested C# helper.
  • Trigger and time a failover safely.

Prerequisites

Why this matters in banking

Harbourline's architecture review says "everything is Multi-AZ". An engineer then tests it by disabling one AZ's networking in a pre-production account, and three things break:

  1. The one NAT gateway was in the failed AZ, so every task in the other two AZs lost outbound internet access and could not reach a third-party sanctions API.
  2. The Aurora cluster had one writer and no reader. Failover needed a brand-new instance to be created, taking minutes instead of seconds.
  3. A background job ran as a singleton on one task; it happened to be on the failed AZ and silently stopped for forty minutes.

"Multi-AZ" is a property you verify per component, not a label you apply to an architecture. This lesson gives you the checklist.

The idea in plain language

Multi-AZ means a service keeps working if one Availability Zone fails, because the service has capacity or copies of data in at least one other AZ. There are three flavours, and it matters which one you are getting:

FlavourWhat it meansExamples
Built in, regional by designThe service spans AZs and you cannot choose otherwiseDynamoDB, S3 Standard, SQS, SNS, EventBridge, Lambda, Step Functions, API Gateway
Optional, you switch it onStandby or replicas exist in another AZ only if you configure themRDS Multi-AZ, ElastiCache Multi-AZ, EFS regional vs One Zone, Aurora readers
Your designYou must place and balance the capacityEC2 and ECS capacity, NAT gateways, your own caches, your background jobs

Analogy: the bank keeps its records in a vault. Some vaults come with a duplicate vault in another building automatically (built in). Some offer one if you pay and ask (optional). And your own staff rota across buildings is a thing you design (your design). A review that says "we are in two buildings" must check each of the three.

What each service does

Amazon RDS (SQL Server, PostgreSQL, MySQL and others) has two Multi-AZ shapes:

  • Multi-AZ DB instance deployment: one primary and one standby in another AZ. The standby is not readable. Replication to it is synchronous at the storage or engine level (the mechanism varies by engine). On failure, RDS flips the DNS name of the instance endpoint to the standby. Failover typically takes 60 to 120 seconds, as the earlier databases lesson explains; measure yours.
  • Multi-AZ DB cluster deployment: one writer and two readable standbys in three AZs, using the engine's native semi-synchronous replication (a commit needs acknowledgement from at least one reader). The readers serve read traffic and are failover targets. Failover first resolves replica lag, then promotes, so lag affects failover time. The documentation lists supported engines, instance classes and Regions, which are limited ⚠ VERIFY before choosing this for SQL Server. It is not the same thing as Aurora.

Amazon Aurora separates compute from storage. The cluster volume keeps six copies of the data across three AZs, and writes are replicated across AZs synchronously, regardless of how many DB instances you run. That protects the data even with a single instance. For availability of the instances:

  • If the cluster has one or more Aurora Replicas, a replica is promoted on failure. AWS documents service as typically restored in less than 60 seconds, often less than 30.
  • If the cluster has no replicas, Aurora recovers the primary host in the same AZ, which AWS documents as typically less than 10 minutes. If the AZ itself is down, you must create an instance in another AZ yourself.
  • You can set promotion tiers (0 highest priority to 15) to choose which replica is promoted.
  • Connect to the cluster endpoint (always the current writer) and the reader endpoint (load-balances readers). RDS Proxy can bypass DNS caching to shorten failover, and preserves connections.

Amazon ElastiCache (Valkey, Redis OSS) with Multi-AZ enabled spreads replicas across AZs and enables automatic failover: on primary failure the replica with the least replication lag is promoted and the primary endpoint's DNS is updated. Writes typically resume within seconds. Replication is asynchronous, so a small amount of data can be lost, which is fine for a cache and a reason not to treat a cache as a system of record. You need at least one replica in a different AZ. A customer-initiated reboot of a primary does not trigger failover.

DynamoDB, S3, SQS, SNS, EventBridge: replicated across multiple AZs by the service. There is no setting to enable. S3 One Zone storage classes and EFS One Zone are the exceptions where you choose a single AZ.

ECS and Fargate: you give the service subnets in multiple AZs, and ECS spreads tasks across them (with AZ rebalancing available). EC2 launch type capacity spreads via the Auto Scaling group's subnets. Spread only works if each AZ has subnets, capacity and quota.

Lambda: runs across AZs automatically. If the function is attached to a VPC, give it subnets in at least two AZs.

Zonal resources that you place yourself:

ResourceZonal?What to do
NAT gatewayYes: lives in one subnet, one AZOne per AZ, with each AZ's route table pointing to its own
Interface VPC endpointsOne network interface per AZ you selectSelect every AZ you use
EC2 instances, EBS volumesYesSpread instances; EBS is tied to one AZ, so use snapshots or replicate at the application layer
SubnetsExactly one AZ eachCreate one per tier per AZ
Load balancer nodesOne per enabled AZEnable all AZs you use; no empty subnets

Your own .NET services: the stateless rules

A service survives an AZ loss only if any copy can serve any request:

  • No local state that matters. Sessions, rate-limit counters, idempotency records and caches that must be shared belong in DynamoDB, ElastiCache or the database.
  • Share ASP.NET Core Data Protection keys. Otherwise a cookie or antiforgery token issued by one task cannot be read by another.
  • Singleton background work needs coordination. If a hosted service must run once, use a lease (conditional write in DynamoDB), a queue consumer, or a scheduled service such as EventBridge Scheduler. "One task runs it" is a single point of failure.
  • Idempotent operations. After a failover, a client or the platform may retry. Money movements need idempotency keys.
  • Reconnect gracefully. Connection pools hold connections to the old primary. They break and must be re-established.
  • Know your AZ. Put the AZ on every log line and metric dimension; during an incident, "all errors come from one AZ" is the fastest diagnosis.

What problem the AWS service solves

Multi-AZ features convert infrastructure failure (a data centre problem) into a brief blip instead of an outage. They do it with no change to your data model, which is why they are the cheapest resilience you can buy.

Honest limits:

  • Failover is not instant. Seconds to minutes, during which writes fail. Your client code must tolerate that.
  • Standbys cost money (a Multi-AZ DB instance costs about double a single-AZ instance ⚠ VERIFY current pricing). Aurora replicas are full instances.
  • It protects against infrastructure loss, not bad data. A bad UPDATE replicates. Use point-in-time recovery.
  • It does not protect against a Region failure. That is DR (The four DR strategies).
  • Failover must be tested. A Multi-AZ configuration that has never failed over is an assumption.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
SQL Server Always On availability group (synchronous replica)RDS Multi-AZ DB instanceManaged; standby not readable; failover by DNS flip; for SQL Server the mechanism depends on edition and engine ⚠ VERIFY
Azure SQL zone-redundant configurationRDS Multi-AZ, or AuroraAurora's storage is replicated across AZs even with one instance
Read-scale Always On secondaryMulti-AZ DB cluster readers, Aurora ReplicasAurora: up to 15 replicas, endpoint load-balances
Azure Cache for Redis zone redundancyElastiCache Multi-AZ with automatic failoverReplicas must exist in other AZs
Zone-redundant App Service / AKS node pools in zonesECS service across subnets in AZs; ASG across subnetsYou choose subnets per AZ; NAT gateways are zonal, whereas Azure NAT Gateway is zonal too, so the same trap applies
Cosmos DB with zone redundancyDynamoDB (always multi-AZ)Nothing to enable
SQL connection resiliency (EnableRetryOnFailure)EF Core with Npgsql EnableRetryOnFailureSame EF Core feature; use it with the cluster or proxy endpoint

Architecture

What happens when AZ a fails:

Security-boundary view: standbys and replicas must have the same controls as the primary. Encryption must be on (a KMS key available in the AZ's Region, which is the same Region), security groups identical, and logging enabled. An unencrypted replica or a differently-permissioned standby is a quiet compliance gap.

Step-by-step implementation

Inventory every component and its AZ behaviour

Make a table: component, regional/zonal/optional, current AZ count, what happens on AZ loss, how it was proven. Include the unglamorous ones (NAT gateways, VPC endpoints, schedulers, secrets rotation Lambda, cron-style tasks).

Choose the database shape

For PostgreSQL-compatible OLTP with the shortest failover, choose Aurora with at least one reader in another AZ. For SQL Server (which Harbourline's legacy estate uses), choose RDS Multi-AZ and plan for the 60 to 120 second failover. Use a Multi-AZ DB cluster where the engine supports it and you want readable standbys without Aurora.

Put capacity in every AZ

Three subnets per tier, tasks spread across them, NAT per AZ, endpoints in each AZ. Enable ECS service AZ rebalancing. Use the AZ-loss capacity formula from the scaling lesson.

Make the application failover-tolerant

Use the cluster endpoint (never an instance endpoint), set connection and command timeouts, enable EF Core EnableRetryOnFailure, keep operations idempotent, and consider RDS Proxy to shorten failover and absorb connection storms.

Add AZ awareness to telemetry

Read the AZ from the ECS task metadata endpoint at start-up and add it to log scope and metric dimensions. Verify spread with AzSpreadChecker.

Fail it on purpose

In a non-production account, force a database failover and watch what your service does. Measure write outage seconds. If you never measured it, you do not know it.

AWS Console instructions

  • RDS Multi-AZ instance: Console → RDS → Databases → your database → Modify → Availability and durability → Multi-AZ deployment. Force a failover with Actions → Reboot → Reboot with failover.
  • Aurora: Console → RDS → Databases → your cluster → Add reader (choose a different AZ). Force failover with Actions → Failover.
  • ElastiCache: Console → ElastiCache → Valkey caches (or Redis OSS) → your cache → Modify → Multi-AZ; test with the failover action on the node/shard page.
  • See task AZ spread: Console → ECS → Clusters → your cluster → Services → your service → Tasks; the task detail shows the Availability Zone.

Console labels move; search the page for "Multi-AZ" or "failover" if a label differs ⚠ VERIFY.

AWS CLI

PROFILE=learn-sandbox
REGION=us-east-1

# Which AZ is each Aurora instance in, and which one is the writer?
aws rds describe-db-clusters --db-cluster-identifier hl-m12-demo --profile $PROFILE --region $REGION \
  --query "DBClusters[0].DBClusterMembers[].{Instance:DBInstanceIdentifier,Writer:IsClusterWriter,Tier:PromotionTier}" --output table
aws rds describe-db-instances --profile $PROFILE --region $REGION \
  --query "DBInstances[].{Id:DBInstanceIdentifier,AZ:AvailabilityZone,MultiAZ:MultiAZ,Engine:Engine}" --output table

# Force an RDS Multi-AZ failover (reboot with failover) and an Aurora failover. Non-production only.
aws rds reboot-db-instance --db-instance-identifier hl-m12-demo-sql --force-failover --profile $PROFILE --region $REGION
aws rds failover-db-cluster --db-cluster-identifier hl-m12-demo --profile $PROFILE --region $REGION

# ElastiCache: confirm Multi-AZ and automatic failover, then test a failover (limited per 24 hours).
aws elasticache describe-replication-groups --replication-group-id hl-m12-cache --profile $PROFILE --region $REGION \
  --query "ReplicationGroups[0].{MultiAZ:MultiAZ,AutoFailover:AutomaticFailover,Status:Status}"
aws elasticache test-failover --replication-group-id hl-m12-cache --node-group-id 0001 --profile $PROFILE --region $REGION

# Count NAT gateways per AZ (you want one per AZ you use) and look for single-AZ interface endpoints.
aws ec2 describe-nat-gateways --filter Name=state,Values=available --profile $PROFILE --region $REGION \
  --query "NatGateways[].{Id:NatGatewayId,Subnet:SubnetId}" --output table
aws ec2 describe-vpc-endpoints --profile $PROFILE --region $REGION \
  --query "VpcEndpoints[?VpcEndpointType=='Interface'].{Svc:ServiceName,Subnets:SubnetIds}" --output json

The identifiers above are examples; the lab stack does not create a database, so use the Module 6 lab database or a small database you create and delete in the same hour.

Infrastructure as Code

The lab stack applies the Multi-AZ rules that do not need a database: three AZs, tasks spread across them, a load balancer in all three and ECS AZ rebalancing. For the database part, the pattern is below.

// Aurora PostgreSQL with a writer plus a reader in another AZ (illustrative excerpt).
var cluster = new DatabaseCluster(this, "Aurora", new DatabaseClusterProps
{
    Engine = DatabaseClusterEngine.AuroraPostgres(new AuroraPostgresClusterEngineProps
    {
        Version = AuroraPostgresEngineVersion.VER_16_6, // use a version your Region supports ⚠ VERIFY
    }),
    Vpc = vpc,
    VpcSubnets = new SubnetSelection { SubnetType = SubnetType.PRIVATE_ISOLATED },
    Writer = ClusterInstance.Provisioned("writer", new ProvisionedClusterInstanceProps
    {
        InstanceType = InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE),
        PromotionTier = 0,
    }),
    Readers =
    [
        ClusterInstance.Provisioned("reader1", new ProvisionedClusterInstanceProps
        {
            InstanceType = InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE),
            PromotionTier = 0, // first choice for promotion
        }),
    ],
    StorageEncrypted = true,
    DeletionProtection = true,
    BackupRetention = Duration.Days(14),
});
// CDK spreads the instances across the AZs of the chosen subnets. Confirm with describe-db-instances.

These database excerpts are illustrative and are not part of the lab stack, to keep the lab cheap ⚠ VERIFY argument names against your CDK and provider versions before using them.

Complete code

Two helpers from the lab, EcsAvailabilityZoneProvider and AzSpreadChecker (both covered by AzAwarenessTests), and how a service uses them with EF Core.

src/Harbourline.Resilience/AzAwareness.cs (excerpt)
public sealed class EcsAvailabilityZoneProvider(HttpClient http, string? metadataUri)
{
    public const string Unknown = "unknown";

    public async Task<string> GetAvailabilityZoneAsync(CancellationToken cancellationToken = default)
    {
        if (string.IsNullOrWhiteSpace(metadataUri))
        {
            return Unknown; // not on ECS (a laptop, a test)
        }

        try
        {
            using var response = await http.GetAsync($"{metadataUri.TrimEnd('/')}/task", cancellationToken);
            response.EnsureSuccessStatusCode();

            await using var body = await response.Content.ReadAsStreamAsync(cancellationToken);
            using var document = await JsonDocument.ParseAsync(body, cancellationToken: cancellationToken);

            return document.RootElement.TryGetProperty("AvailabilityZone", out var az) && az.GetString() is { Length: > 0 } name
                ? name
                : Unknown;
        }
        catch (HttpRequestException) { return Unknown; } // diagnostics must never take the service down
        catch (JsonException) { return Unknown; }
    }
}

public static class AzSpreadChecker
{
    public sealed record Result(IReadOnlyDictionary<string, int> TasksPerAz, int Skew, bool MeetsMinimumAzs, bool IsBalanced);

    public static Result Evaluate(IEnumerable<string> azPerTask, int expectedAzCount)
    {
        var counts = azPerTask.GroupBy(az => az).ToDictionary(g => g.Key, g => g.Count());
        var populated = counts.Count;
        var highest = counts.Count == 0 ? 0 : counts.Values.Max();
        var lowest = populated < expectedAzCount || counts.Count == 0 ? 0 : counts.Values.Min();
        var skew = highest - lowest;

        return new Result(counts, skew, populated >= expectedAzCount, skew <= 1);
    }
}
Program.cs (Accounts API excerpt)
using Harbourline.Resilience;
using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

// The cluster endpoint always points at the current writer; never use an instance endpoint.
var connectionString = builder.Configuration.GetConnectionString("Accounts")
    ?? throw new InvalidOperationException("ConnectionStrings:Accounts is required.");

builder.Services.AddDbContext<AccountsDbContext>(options =>
    options.UseNpgsql(connectionString, npgsql =>
    {
        // Retry transient errors, including the connection errors seen during a failover.
        npgsql.EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(10), errorCodesToAdd: null);
        npgsql.CommandTimeout(15);
    }));

// Find out which AZ this task runs in, once, and add it to every log line.
var azProvider = new EcsAvailabilityZoneProvider(
    new HttpClient { Timeout = TimeSpan.FromSeconds(2) },
    Environment.GetEnvironmentVariable("ECS_CONTAINER_METADATA_URI_V4"));
var availabilityZone = await azProvider.GetAvailabilityZoneAsync();

var app = builder.Build();

app.Use(async (context, next) =>
{
    using (app.Logger.BeginScope(new Dictionary<string, object> { ["az"] = availabilityZone }))
    {
        await next(context);
    }
});

app.MapGet("/accounts/{id}", async (string id, AccountsDbContext db, CancellationToken ct) =>
    await db.Accounts.FindAsync([id], ct) is { } account ? Results.Ok(account) : Results.NotFound());

app.Run();

public sealed class AccountsDbContext(DbContextOptions<AccountsDbContext> options) : DbContext(options)
{
    public DbSet<Account> Accounts => Set<Account>();
}

public sealed record Account(string AccountId, string Currency, decimal Balance);

The second excerpt is illustrative and needs the Npgsql.EntityFrameworkCore.PostgreSQL package and a mapped Account key; the lab tests cover the helpers, not the web host.

Code explained line by line

  1. ECS_CONTAINER_METADATA_URI_V4 is an environment variable that ECS sets on each container (Fargate platform version 1.4.0 and later). Its /task path returns JSON that includes an AvailabilityZone property.
  2. EcsAvailabilityZoneProvider returns unknown for anything abnormal (not on ECS, endpoint error, malformed JSON): a diagnostics helper must never be the reason a service fails to start.
  3. AzSpreadChecker.Evaluate counts tasks per AZ. An expected AZ with no tasks counts as zero, so "all tasks in one AZ" has a skew equal to the task count, and a one-task difference between AZs is accepted as balanced.
  4. EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: 10 s) turns EF Core's execution strategy on, retrying transient failures with increasing delay. Together with a cluster endpoint, this rides out a typical failover. A non-idempotent write inside a retry can run twice if the first attempt committed but the response was lost, so use transactions and idempotency keys for money movements.
  5. CommandTimeout(15) bounds one command, so a hung connection to the old writer does not pin a request for minutes.
  6. The app.Use middleware puts az into the logging scope, so every log line from a request carries it.

Security considerations

  • Encrypt every replica and standby with KMS, and keep the same security groups and IAM database authentication posture. Aurora and RDS Multi-AZ inherit encryption from the primary, but verify.
  • Secrets across failover. Use Secrets Manager with managed master credentials, and make sure the application reads the secret at connection time (or refreshes), so a rotation or failover does not strand it.
  • RDS Proxy adds an IAM and secret-based authentication path; scope its role to the single secret.
  • Do not expose instance endpoints. They bypass failover and tempt people to hard-code a host.
  • Failover testing is a change. Do it through a controlled, ticketed process, with CloudTrail evidence, never in production without approval.

⚠ Learning shortcut — not for production

The lab and these excerpts use small instance classes, a single secret and DeletionProtection shown for illustration. They tolerate an AZ failure but are not a performance or compliance design. In production, size from load tests, add RDS Proxy where connection counts are high, turn on audit logging, set backup retention to your policy, and have compliance and the data-protection officer review where the data may live (UK GDPR) before choosing Regions.

Banking use case

Harbourline's Accounts API reads and writes an Aurora PostgreSQL cluster, caches balances in ElastiCache for Valkey and sends account events to SQS. The Multi-AZ review table, abbreviated:

ComponentAZ behaviourEvidence
ALB and ECS tasks3 AZs, 18 tasks (6 per AZ), AZ rebalancing onSpread check in CI, drill
AuroraWriter plus two readers across 3 AZs, promotion tier 0 on one readerFailover drill: 24 s measured outage
ValkeyPrimary plus two replicas in 3 AZs, Multi-AZ ontest-failover event log
SQS, DynamoDB idempotency tableRegional by designAWS documentation
NAT gatewaysOne per AZIaC test
Nightly interest accrual jobScheduled by EventBridge Scheduler, queue-driven workersNot tied to one task

The numbers are fictional teaching values.

Common mistakes

  • One NAT gateway for the VPC. An AZ failure takes outbound access away from other AZs.
  • Aurora with no reader and the belief that "Aurora is Multi-AZ". The storage is; the compute may not be.
  • Using instance endpoints. They do not follow failover.
  • Treating a cache as a database. Replication to the replica is asynchronous.
  • Retrying non-idempotent writes without idempotency keys.
  • Long DNS caching or connection pools that never recycle. The app keeps talking to the old writer. Set sensible pool lifetimes and use the cluster endpoint or RDS Proxy.
  • Singleton jobs. "Runs on task 1" is not a design.
  • Never testing failover. The first test is then an incident.

Troubleshooting

SymptomLikely causeFix
Writes fail for minutes after Aurora failoverNo reader to promote, or app caching the old IP/DNSAdd a reader in another AZ; use the cluster endpoint; set pool lifetime; consider RDS Proxy
After failover the app throws read-only transaction errorsConnected through an old connection to a demoted nodeRetry; recycle connections; use cluster endpoint
RDS Multi-AZ failover takes longer than expectedLarge crash recovery, long-running transactionsMeasure, keep transactions short, consider Aurora
Tasks all in one AZSingle subnet, capacity shortage, rebalancing offAdd subnets; enable rebalancing; check quotas
Outbound calls fail from two AZs when one AZ has an incidentSingle NAT gatewayOne NAT per AZ with per-AZ route tables
ElastiCache test-failover fails or is rejectedMulti-AZ/automatic failover not enabled, or limit of 15 shard tests per 24 hoursEnable Multi-AZ; stay within the limit
AzSpreadChecker shows skew of 2 or moreRebalancing lag or an AZ shortageCheck ECS service events and subnet IP space

Cost and cleanup

Reading the lesson and running the helpers locally costs nothing. If you experiment with real databases, an Aurora cluster with a writer and reader costs several dollars per day even at small sizes ⚠ VERIFY current prices; an RDS Multi-AZ instance costs about twice a single-AZ instance; ElastiCache nodes bill hourly. Create, test failover, and delete within an hour, and take no final snapshot unless you want one.

# Delete a lab Aurora cluster (instances first, then cluster) with no final snapshot. Non-production only.
aws rds delete-db-instance --db-instance-identifier hl-m12-demo-0 --skip-final-snapshot --profile learn-sandbox --region us-east-1
aws rds delete-db-instance --db-instance-identifier hl-m12-demo-1 --skip-final-snapshot --profile learn-sandbox --region us-east-1
aws rds delete-db-cluster --db-cluster-identifier hl-m12-demo --skip-final-snapshot --profile learn-sandbox --region us-east-1

# Confirm nothing tagged for this lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
  --profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"

If deletion protection is on, disable it first (modify-db-cluster --no-deletion-protection).

Exercises

  • 🟢 Guided: Run AzAwarenessTests. Then write a table of your own account's resources (or the Harbourline estate in this lesson) with columns regional, optional, zonal, and mark every zonal one.
  • 🟡 Challenge: An architecture has: 3-AZ ECS service, one NAT gateway, an RDS Multi-AZ SQL Server instance, an ElastiCache Valkey cluster with one node, a nightly job on one task. List every component that fails the "survives an AZ loss" test and give the fix and its approximate cost direction.
  • 🔴 Live problem: "We forced an Aurora failover in staging and the API returned 500s for 4 minutes, but AWS says under a minute." Investigate: what do you check first, second and third, and what changes do you propose?

Assignment

Write docs/multi-az-review.md for Harbourline's Payments service: a component table (regional, optional, zonal), the AZ-loss behaviour of each, the evidence you would collect, and a failover drill plan with the success criteria (maximum write outage seconds, zero lost acknowledged payments).

Solutions

Knowledge check

Interview questions

  • "Is Aurora Multi-AZ by default?" Its storage is replicated across three AZs by default. Instance-level availability depends on having at least one Aurora Replica in another AZ to promote; otherwise recovery is slower.
  • "How would you prove a service survives an AZ loss?" Inventory zonal components, ensure capacity per AZ meets the AZ-loss formula, then run a game day that removes an AZ (for example with FIS or a zonal shift in pre-production) and measure errors and recovery time.
  • "What is the most common hidden single point of failure in a Multi-AZ VPC?" A single NAT gateway, followed by singleton background jobs and single-replica caches.
  • "How do you make an EF Core app handle failover?" Cluster or proxy endpoint, command timeouts, EnableRetryOnFailure, idempotent writes, pool lifetime tuning, and logs that carry the AZ.

Certification practice

Completion checklist

  • I can say whether Multi-AZ is built in, optional or my responsibility for each service I use
  • I can explain the difference between RDS Multi-AZ instance, Multi-AZ DB cluster and Aurora
  • I ran AzAwarenessTests and understood the spread checker
  • I wrote a Multi-AZ review for one service
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • Multi-AZ is built in for some services, optional for others, and your design for compute, NAT and jobs. Verify each.
  • Aurora protects data across three AZs by default, but you need a reader in another AZ for fast instance failover.
  • Failover takes seconds to minutes; your application must retry and reconnect, and money movements must be idempotent.
  • NAT gateways, singleton jobs and single-node caches are the classic hidden single points of failure.
  • Measure failover. An untested Multi-AZ design is an assumption.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page