Database failover across Regions
Choose and rehearse cross-Region database recovery: Aurora Global Database switchover and failover, DynamoDB global tables (MREC and MRSC), and RDS cross-Region read replica promotion, with a runbook and C# helpers that make the decision measurable.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.RDS, AWSSDK.CloudWatch) · AWS CLI v2 · Aurora PostgreSQL 16/17 · DynamoDB global tables 2019.11.21 · Regions us-east-1 (primary) and us-west-2 (DR). Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the difference between an Aurora Global Database switchover (zero data loss, planned) and a failover (unplanned, possible data loss), and pick the right one for a given incident.
- Compare DynamoDB global tables in MREC (multi-Region eventual consistency) and MRSC (multi-Region strong consistency) modes against an RPO target.
- Promote an RDS cross-Region read replica and list the extra steps that Aurora Global Database does for you.
- Write a database failover runbook with decision points, evidence capture and failback.
- Use C# to read replication lag from CloudWatch and encode the "switchover or failover?" decision as a tested function.
Prerequisites
- Multi-AZ, read replicas, backups and point-in-time recovery, the single-Region protections this lesson builds on.
- Amazon Aurora and DynamoDB fundamentals.
- The earlier lessons in this module, in particular the four DR strategies and the active-active versus active-passive lesson (see the module overview).
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK. Two Regions enabled in your sandbox account (us-east-1,us-west-2).
Why this matters in banking
Harbourline's ledger database holds the bookings behind every balance. The operational-resilience plan says the payments ledger must be back within 30 minutes (RTO) and may lose at most 5 minutes of bookings (RPO) if a whole Region is lost. The engineers have Multi-AZ and nightly backups, so they believe they are covered.
They are not. Multi-AZ protects against a lost Availability Zone, not a lost Region. Restoring last night's backup in another Region means up to a day of lost payments and hours of restore time. To meet 30 minutes and 5 minutes, there has to be a database already running in the second Region, kept current by replication, and a rehearsed procedure to make it the writer. This lesson is that procedure, and the honest discussion of what you can lose while you do it.
Regulators do not mandate a specific technology. They expect you to set an impact tolerance for each important business service and to show evidence that you can stay within it. Replication settings and a timed failover drill are that evidence. Always involve your risk and compliance teams; this lesson does not claim any compliance outcome.
The idea in plain language
Cross-Region recovery has two halves:
- Data: a second copy of the database in another Region that is only seconds behind.
- Control: a decision and a mechanism to make that copy the one that accepts writes, and to send the application to it.
Replication between Regions is asynchronous for most AWS databases. The primary commits a write and tells the client "done" before the remote copy has it. This is what keeps writes fast, and it is also exactly why the RPO is greater than zero: the writes in flight when the Region dies may never arrive.
Analogy: Harbourline's head office phones every booking through to the disaster-recovery branch. Normally a few seconds pass between the booking and the call. If the head office burns down mid-call, the DR branch has everything up to the last completed call. A switchover is closing head office on purpose after the last call has finished, so nothing is lost. A failover is the fire: you take what the DR branch has.
What problem each option solves
| Option | Replication | Typical RPO | Typical RTO | Writers | Key limits |
|---|---|---|---|---|---|
| Aurora Global Database | Storage-level, asynchronous, dedicated infrastructure | Seconds (lag dependent) | Minutes for failover; switchover typically under 30 s on current engine versions | One primary Region; secondaries are read-only | Same major and minor engine versions needed for managed operations ⚠ VERIFY patch-level rules |
| DynamoDB global tables, MREC | Asynchronous, per-item | Replication delay, usually about a second | Seconds to minutes (application redirect) | All Regions accept writes | Last-writer-wins conflicts; transactions atomic only in the source Region |
| DynamoDB global tables, MRSC | Synchronous to at least one other Region before the write returns | Zero | Seconds to minutes (application redirect) | All replicas accept writes | Exactly three Regions (or two plus a witness); no transactions, no TTL, no LSIs; higher write latency |
| RDS cross-Region read replica | Engine-native, asynchronous | Replica lag (seconds to minutes) | Minutes: promote, then repoint | One primary; replica is read-only until promoted | Promotion is one-way: the replica leaves the replication relationship |
| AWS Backup cross-Region copy | Scheduled copies of backups | Hours (backup interval) | Hours (restore) | None until restored | Cheapest; see AWS Backup in depth |
The figures are guidance from the AWS documentation. Your own measured numbers matter more, which is why the drill later in the lesson records them.
Aurora Global Database: switchover versus failover
An Aurora global database has one primary cluster and up to several secondary clusters in other Regions. The secondaries are read-only until promoted. Aurora provides two promotion operations:
- Switchover (previously called "managed planned failover"). Use it when everything is healthy: Region rotation drills, planned maintenance, or failing back after an incident. Aurora waits for the chosen secondary to catch up, makes the old primary read-only, then promotes the secondary. RPO is zero. On current engine versions the read/write interruption is typically under 30 seconds ⚠ VERIFY your engine version qualifies.
- Failover. Use it only for an unplanned Regional outage. It does not wait for replication to catch up, so you can lose the writes that had not yet arrived. In the CLI you must pass
--allow-data-lossto make that explicit. Aurora attempts to snapshot the old primary's storage at the point of failure so that you can later recover missing rows, and attempts to fence writes on the old primary, but fencing is best-effort, which is why split brain (two clusters accepting writes) is a documented risk.
The recommended failover method is the managed one: when the old primary Region comes back, Aurora adds it back as a secondary, restoring the original topology. The alternative manual failover (detach the secondary, which becomes standalone, then build a new global database around it) exists for cases where managed failover is not possible, for example incompatible engine versions.
Regulators and auditors like the switchover for one more reason: the Aurora documentation describes regional rotation (moving the writer to another Region for a period) as a way to prove that disaster-recovery procedures really work. Harbourline's plan uses a quarterly switchover for exactly that.
Controlling RPO (Aurora PostgreSQL). The rds.global_db_rpo cluster parameter lets you declare an RPO in seconds (minimum 20). Aurora then blocks commits on the primary whenever every secondary is further behind than the limit. That turns RPO from a hope into a guarantee, at the price of write stalls when replication is slow. The documentation specifically recommends keeping the parameter at its default in the secondary's parameter group in a two-Region setup, otherwise a failover could pause writes on the newly promoted cluster. Treat it as a deliberate decision with your product owner, not a default.
Application endpoints. Use the global writer endpoint so that the application connection string does not change after a promotion. Keep DNS caching short (the documentation suggests a TTL as low as 5 seconds) and confirm that your pooled connections are recycled, otherwise some instances keep talking to the old writer.
DynamoDB global tables: MREC versus MRSC
A global table is a set of replica tables, one per Region, sharing a name, key schema and data. Version 2019.11.21 is the current version. There are two consistency modes, chosen when you create the table and not changeable afterwards:
- MREC (default). Every replica accepts writes; changes replicate asynchronously, usually within a second. If the same item is written in two Regions at nearly the same time, last writer wins by timestamp. Strongly consistent reads are only strong with respect to writes in the same Region. Transactions (
TransactWriteItems) are atomic only in the Region where they run, so another Region can briefly see half of a transaction. - MRSC. A write is replicated synchronously to at least one other Region before it returns, so RPO is zero and strongly consistent reads return the latest data in any replica. It must have exactly three Regions (three replicas, or two replicas and a witness), does not support transactions, TTL or local secondary indexes, and conflicting concurrent writes to one item in different Regions fail with
ReplicatedWriteConflictException, which you retry. MRSC is only available in a documented list of Regions ⚠ VERIFY the list for your chosen pair.
The headline for a bank: MREC's last-writer-wins resolution and region-local transactions make it unsafe for a ledger that must apply debits and credits exactly once, unless the application is designed so each account is only ever written in one Region at a time (a "home Region per account"). MRSC removes the lost-update problem but not the application work, since you lose transactions. See DynamoDB banking correctness for how to design single-item conditional writes that survive these constraints. That is why the earlier lesson on active-active designs concluded that most ledgers stay single-writer, and why global tables fit better for session data, reference data, preferences and idempotency records.
RDS cross-Region read replicas
For RDS for PostgreSQL, MySQL, MariaDB (and SQL Server Enterprise ⚠ VERIFY edition and version rules), you can create a read replica in another Region. It is asynchronous, uses the engine's native replication, and can lag by seconds to minutes under load. To recover you promote the replica to a standalone instance, then do everything Aurora would do for you: repoint the application, fix parameter groups, create replicas of the new primary, and rebuild the old primary as a replica after the Region returns. Promotion is not reversible, and there is no managed switchback: failback means building a new replica in the original Region and promoting that.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure SQL Database active geo-replication and failover groups | Aurora Global Database (failover and switchover) | Aurora's replication runs at the storage layer; there is a dedicated global writer endpoint, like a failover group listener |
| SQL Server Always On availability groups across data centres | RDS cross-Region read replica promotion | RDS promotion is a one-way operation; you rebuild the replica afterwards |
Planned vs forced failover in failover groups (allow data loss) | Switchover vs failover with --allow-data-loss | The same trade-off, made explicit in the CLI flag |
| Cosmos DB multi-region writes | DynamoDB global tables (MREC) | Last-writer-wins by default; there is no custom conflict policy |
| Cosmos DB strong consistency, multi-region (not generally available for multi-write) | DynamoDB MRSC | Synchronous cross-Region commit at the cost of latency and transactions |
| Azure Traffic Manager health probes redirecting clients | Route 53 failover records plus application logic | The database role change and the traffic change are two separate steps you must coordinate |
Architecture
The sequence of a failover (unplanned) differs from a switchover (planned):
Security-boundary view: the failover action is a powerful, rare, high-impact permission. Put it behind a dedicated break-glass role in the management plane, require MFA, send CloudTrail events to a separate log account, and keep the DR Region's KMS keys, security groups and secrets replicated before the disaster. Secrets Manager secrets and KMS multi-Region keys need to be replicated ahead of time because you cannot create them in a hurry during an outage in the home Region.
Step-by-step implementation
This is a design and drill sequence. Run steps 1 to 3 in a sandbox only if you accept the cost shown later; otherwise do them as a paper exercise.
Set the targets and choose the mechanism
Write down the RTO and RPO per data store. For the Harbourline ledger: RTO 30 minutes, RPO 5 minutes. Aurora Global Database (seconds of lag, failover in minutes) meets that. For the idempotency-key table, DynamoDB MREC with a single home Region per key meets it cheaply.
Build the global database
Create the primary cluster, a global cluster that wraps it, and a secondary cluster in the DR Region with at least one instance. A secondary cluster with no instance (headless) must be given an instance before it can be promoted. Use the IaC below.
Replicate the things that are not in the database
Replicate secrets (Secrets Manager replica secrets), use multi-Region KMS keys or a separate key per Region, copy parameter groups and alarms, and make sure the DR Region has subnets, security groups and an application stack ready (a warm standby or pilot light).
Alarm on replication lag
Create a CloudWatch alarm on AuroraGlobalDBRPOLag for the secondary, with a threshold at half your RPO. Replication lag is the RPO you would actually get today.
Rehearse a switchover, then a failover
Do the switchover first: it is safe and it measures your real application recovery time. Rehearse the failover as a paper exercise with a stopwatch, or in a throwaway environment. Record times for detect, decide, promote, repoint, verify.
Fail back
Failback is a second change, with its own risk. After a failover, wait until Aurora has rebuilt the old primary as a secondary, check lag, then run a switchover during a quiet period.
AWS Console instructions
- Aurora Global Database: Console → RDS → Databases → select the cluster → Actions → Add AWS Region to create a secondary. To promote, select the global database → Actions → Switch over or fail over global database and choose Switchover or Failover (allow data loss). Failover asks you to type
confirm. - Replication lag: Console → CloudWatch → Metrics → RDS → Per-Cluster Metrics →
AuroraGlobalDBRPOLag(milliseconds). Check it from the current primary's view. - DynamoDB global table: Console → DynamoDB → Tables → your table → Global tables tab → Create replica and pick the Region. For MRSC choose the consistency mode while creating the table's replicas ⚠ VERIFY exact console labels.
- RDS read replica: Console → RDS → Databases → select the instance → Actions → Create read replica, and choose the destination Region. To promote: select the replica → Actions → Promote.
AWS CLI
# Check how far behind each secondary is (run against the primary's Region).
# AuroraGlobalDBRPOLag is in milliseconds.
aws cloudwatch get-metric-statistics \
--namespace AWS/RDS --metric-name AuroraGlobalDBRPOLag \
--dimensions Name=DBClusterIdentifier,Value=hl-m12-ledger-primary \
--start-time "$(date -u -d '15 minutes ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
--period 60 --statistics Maximum \
--region us-east-1 --profile learn-sandbox
# PLANNED: zero data loss. Run with the Region of the CURRENT primary.
aws rds switchover-global-cluster \
--global-cluster-identifier hl-m12-ledger-global \
--target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:hl-m12-ledger-secondary \
--region us-east-1 --profile learn-sandbox
# UNPLANNED: may lose recent writes. Run with the Region of the SECONDARY you are promoting.
aws rds failover-global-cluster \
--global-cluster-identifier hl-m12-ledger-global \
--target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:hl-m12-ledger-secondary \
--allow-data-loss \
--region us-west-2 --profile learn-sandbox
# Look for the system snapshot Aurora tries to take of the old primary after a failover.
aws rds describe-db-cluster-snapshots \
--snapshot-type awsbackup --query "DBClusterSnapshots[?contains(DBClusterSnapshotIdentifier,'unplanned-global-failover')]" \
--region us-west-2 --profile learn-sandbox
# DynamoDB: list replicas and their status for a global table.
aws dynamodb describe-table --table-name hl-m12-idempotency \
--query "Table.Replicas[].{Region:RegionName,Status:ReplicaStatus}" \
--region us-east-1 --profile learn-sandbox
# RDS (non-Aurora): promote a cross-Region read replica. One-way operation.
aws rds promote-read-replica --db-instance-identifier hl-m12-ledger-replica \
--region us-west-2 --profile learn-sandboxThe account number 111122223333 is a placeholder. Note that the switchover is issued in the primary's Region and the failover in the secondary's Region, matching the AWS documentation. The describe-db-cluster-snapshots snapshot-type filter above is a best guess for the system snapshot ⚠ VERIFY, so if it returns nothing, list all snapshots for the cluster and look for the rds:unplanned-global-failover- prefix.
Infrastructure as Code
A global database spans two Regions, so it uses two stacks. The primary stack uses the high-level DatabaseCluster and wraps it in a CfnGlobalCluster; the secondary stack uses the low-level CfnDBCluster with the same global identifier. Cross-Region encryption needs a KMS key in each Region.
using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.RDS;
using Constructs;
namespace Harbourline.Ha.Infra;
public sealed class LedgerPrimaryStack : Stack
{
public const string GlobalId = "hl-m12-ledger-global";
public DatabaseCluster Cluster { get; }
public LedgerPrimaryStack(Construct scope, string id, IVpc vpc, IStackProps props)
: base(scope, id, props)
{
var key = new Key(this, "DbKey", new KeyProps { EnableKeyRotation = true });
Cluster = new DatabaseCluster(this, "Primary", new DatabaseClusterProps
{
ClusterIdentifier = "hl-m12-ledger-primary",
Engine = DatabaseClusterEngine.AuroraPostgres(new AuroraPostgresClusterEngineProps
{
// Pick a version that supports global databases in both Regions. ⚠ VERIFY
Version = AuroraPostgresEngineVersion.VER_16_8
}),
Vpc = vpc,
VpcSubnets = new SubnetSelection { SubnetType = SubnetType.PRIVATE_ISOLATED },
StorageEncrypted = true,
StorageEncryptionKey = key,
Writer = ClusterInstance.Provisioned("writer", new ProvisionedClusterInstanceProps
{
InstanceType = Amazon.CDK.AWS.EC2.InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE)
}),
Readers = new[]
{
ClusterInstance.Provisioned("reader", new ProvisionedClusterInstanceProps
{
InstanceType = Amazon.CDK.AWS.EC2.InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE)
})
},
Backup = new BackupProps { Retention = Duration.Days(14) },
DeletionProtection = false, // learning shortcut: see Security considerations
RemovalPolicy = RemovalPolicy.DESTROY
});
// Wrap the existing primary cluster in a global cluster.
_ = new CfnGlobalCluster(this, "Global", new CfnGlobalClusterProps
{
GlobalClusterIdentifier = GlobalId,
SourceDbClusterIdentifier = Cluster.ClusterArn
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m12-database-failover");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
}
}
public sealed class LedgerSecondaryStack : Stack
{
public LedgerSecondaryStack(Construct scope, string id, string subnetGroupName,
string securityGroupId, IStackProps props)
: base(scope, id, props)
{
// A separate key in the DR Region. Replicated storage is re-encrypted with it.
var key = new Key(this, "DbKey", new KeyProps { EnableKeyRotation = true });
var secondary = new CfnDBCluster(this, "Secondary", new CfnDBClusterProps
{
DbClusterIdentifier = "hl-m12-ledger-secondary",
GlobalClusterIdentifier = LedgerPrimaryStack.GlobalId,
Engine = "aurora-postgresql",
EngineVersion = "16.8", // must match the primary's major and minor version
DbSubnetGroupName = subnetGroupName,
VpcSecurityGroupIds = new[] { securityGroupId },
StorageEncrypted = true,
KmsKeyId = key.KeyArn
// No MasterUsername or password: a secondary inherits data from the primary.
});
// A secondary needs at least one instance before it can be promoted.
_ = new CfnDBInstance(this, "SecondaryInstance", new CfnDBInstanceProps
{
DbClusterIdentifier = secondary.Ref,
DbInstanceClass = "db.r6g.large",
Engine = "aurora-postgresql"
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m12-database-failover");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
}
}DynamoDB global table with the L2 TableV2 construct, which creates a global table with replicas:
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Constructs;
namespace Harbourline.Ha.Infra;
public sealed class IdempotencyTableStack : Stack
{
public IdempotencyTableStack(Construct scope, string id, IStackProps props)
: base(scope, id, props)
{
_ = new TableV2(this, "Idempotency", new TablePropsV2
{
TableName = "hl-m12-idempotency",
PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
Billing = Billing.OnDemand(),
TimeToLiveAttribute = "expiresAt",
PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification
{
PointInTimeRecoveryEnabled = true
},
// Replicas in other Regions. The stack's own Region is the first replica.
Replicas = new[] { new ReplicaTableProps { Region = "us-west-2" } },
RemovalPolicy = RemovalPolicy.DESTROY
});
// This is MREC (the default). MRSC needs three Regions and a consistency
// setting at creation. ⚠ VERIFY the TablePropsV2 property name in your CDK version.
}
}Complete code
Two small, testable pieces. First, a pure function that turns the facts you can measure into a recommendation, so the logic is reviewed and unit-tested rather than improvised at 03:00.
namespace Harbourline.Ha.Failover;
public enum Recommendation
{
Switchover, // planned, zero data loss
FailoverAllowDataLoss,
WaitAndRetry,
DoNotPromote
}
public sealed record FailoverFacts(
bool PrimaryRegionHealthy,
bool ApplicationHealthy,
TimeSpan LagOfBestSecondary,
TimeSpan RpoTarget,
bool SecondaryEngineVersionMatchesPrimary,
bool TwoApproversRecorded);
public static class FailoverAdvisor
{
public static (Recommendation Action, string Reason) Advise(FailoverFacts f)
{
if (!f.TwoApproversRecorded)
return (Recommendation.DoNotPromote, "A disaster declaration needs two recorded approvers.");
if (f.PrimaryRegionHealthy)
{
// Everything works: only a planned switchover is appropriate.
return f.SecondaryEngineVersionMatchesPrimary
? (Recommendation.Switchover, "Primary is healthy; use switchover for zero data loss.")
: (Recommendation.DoNotPromote, "Engine versions differ; managed operations need matching versions.");
}
// Primary Region is impaired from here on.
if (f.ApplicationHealthy)
return (Recommendation.WaitAndRetry, "Application still serves traffic; keep watching before declaring.");
return f.LagOfBestSecondary <= f.RpoTarget
? (Recommendation.FailoverAllowDataLoss,
$"Best secondary lag {f.LagOfBestSecondary.TotalSeconds:F0}s is within the {f.RpoTarget.TotalSeconds:F0}s RPO.")
: (Recommendation.FailoverAllowDataLoss,
$"Lag {f.LagOfBestSecondary.TotalSeconds:F0}s exceeds the RPO: the business must accept the data loss explicitly.");
}
}Second, a console tool that reads real lag from CloudWatch and executes the AWS call only when you pass --execute.
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.RDS;
using Amazon.RDS.Model;
using Harbourline.Ha.Failover;
var globalId = args.ElementAtOrDefault(0) ?? "hl-m12-ledger-global";
var primaryClusterId = args.ElementAtOrDefault(1) ?? "hl-m12-ledger-primary";
var targetArn = args.ElementAtOrDefault(2)
?? throw new ArgumentException("Usage: <globalId> <primaryClusterId> <targetClusterArn> [--execute]");
var execute = args.Contains("--execute");
using var cw = new AmazonCloudWatchClient(RegionEndpoint.USEast1);
// 1. Read the worst lag over the last 5 minutes (milliseconds).
var stats = await cw.GetMetricStatisticsAsync(new GetMetricStatisticsRequest
{
Namespace = "AWS/RDS",
MetricName = "AuroraGlobalDBRPOLag",
Dimensions = [new Dimension { Name = "DBClusterIdentifier", Value = primaryClusterId }],
StartTime = DateTime.UtcNow.AddMinutes(-5),
EndTime = DateTime.UtcNow,
Period = 60,
Statistics = ["Maximum"]
});
var lagMs = stats.Datapoints.Count == 0 ? double.NaN : stats.Datapoints.Max(d => d.Maximum ?? 0);
Console.WriteLine($"Worst replication lag (5 min): {lagMs:F0} ms");
var facts = new FailoverFacts(
PrimaryRegionHealthy: args.Contains("--primary-healthy"),
ApplicationHealthy: args.Contains("--app-healthy"),
LagOfBestSecondary: double.IsNaN(lagMs) ? TimeSpan.MaxValue : TimeSpan.FromMilliseconds(lagMs),
RpoTarget: TimeSpan.FromMinutes(5),
SecondaryEngineVersionMatchesPrimary: true,
TwoApproversRecorded: args.Contains("--two-approvers"));
var (action, reason) = FailoverAdvisor.Advise(facts);
Console.WriteLine($"Advice: {action}. {reason}");
if (!execute || action is Recommendation.WaitAndRetry or Recommendation.DoNotPromote)
{
Console.WriteLine("Dry run: nothing changed. Pass --execute to act on the advice.");
return;
}
// 2. Switchover is issued in the primary's Region, failover in the secondary's Region.
// The target ARN tells us the secondary's Region.
var targetRegion = RegionEndpoint.GetBySystemName(targetArn.Split(':')[3]);
using var rds = new AmazonRDSClient(action == Recommendation.Switchover ? RegionEndpoint.USEast1 : targetRegion);
if (action == Recommendation.Switchover)
{
await rds.SwitchoverGlobalClusterAsync(new SwitchoverGlobalClusterRequest
{
GlobalClusterIdentifier = globalId,
TargetDbClusterIdentifier = targetArn
});
}
else
{
await rds.FailoverGlobalClusterAsync(new FailoverGlobalClusterRequest
{
GlobalClusterIdentifier = globalId,
TargetDbClusterIdentifier = targetArn,
AllowDataLoss = true
});
}
Console.WriteLine($"Requested {action} at {DateTime.UtcNow:O}. Record this timestamp in the drill log.");Package references: AWSSDK.RDS and AWSSDK.CloudWatch 4.x (look up the exact versions on NuGet). The request property names (TargetDbClusterIdentifier, AllowDataLoss) follow the SDK's PascalCase mapping of the API fields ⚠ VERIFY against the SDK version you install.
Code explained line by line
FailoverFactscollects the observable inputs (health, lag, approvals). Making them a record means a test can construct any scenario without AWS.Advisechecks approvals first: a failover is a business decision, so the code refuses to recommend promotion without two recorded approvers.- If the primary Region is healthy, the only legitimate operation is a switchover, which waits for replication to finish. This prevents someone running the data-loss path during a drill.
- If the primary is impaired but the application still serves traffic, the advisor says wait. Declaring a disaster too early is itself a risk.
- When lag exceeds the RPO, the advisor still returns the failover but with a reason that spells out that data loss exceeds target, so the person approving cannot miss it.
- In
Program.cs,GetMetricStatisticsAsyncreadsAuroraGlobalDBRPOLagasMaximumover 5 minutes. The maximum is used because the worst lag in the window is the realistic data-loss figure. - The tool is dry run by default. Requiring
--executemakes accidental failovers during a demo much less likely. - The RDS client is created in the primary's Region for a switchover and in the target's Region for a failover, mirroring the AWS guidance about which Region to call.
- The last line prints a timestamp: your RTO measurement starts from the moment the decision was acted on.
Security considerations
⚠ Learning shortcut — not for production
The IaC above sets DeletionProtection = false, skip_final_snapshot = true and RemovalPolicy.DESTROY so you can tear down the lab. In production, keep deletion protection on, take final snapshots, keep backups in a Vault Lock vault in a separate account, and use manage_master_user_password (Secrets Manager) with replicated secrets. Never put a password in IaC variables.
- Least privilege for the failover action.
rds:FailoverGlobalClusterandrds:SwitchoverGlobalClusterbelong to a break-glass role, assumed with MFA, with session tags recording the incident number. Everyone else gets read-only. - Encryption. Each Region needs a usable KMS key. A global database encrypts storage in each Region with a key in that Region. Verify that the DR Region's key policy allows the application role and the RDS service.
- Secrets and configuration. Replicate Secrets Manager secrets and Parameter Store values to the DR Region ahead of time. A failover that cannot read the database password is not a failover.
- Data residency. Replicating a UK customer ledger to a US Region is a transfer of personal data. For Harbourline's real design, the DR Region for UK data would be another UK or EEA Region.
us-west-2is used here for lab convenience only. Ask your data-protection officer; see UK GDPR and data residency. - Split brain. Two clusters accepting writes means two versions of the truth, which for a bank is a reconciliation incident. Stop application writes before the promotion, use the global writer endpoint and short DNS caching, and keep the old primary's system snapshot so you can reconcile.
- Evidence. CloudTrail logs the API calls. Store the drill log (timestamps, approvers, lag, results) with your operational-resilience evidence.
Banking use case
Harbourline's resilience plan assigns each datastore a tier:
| Datastore | Tier | Mechanism | RTO / RPO |
|---|---|---|---|
| Ledger (PostgreSQL) | Critical | Aurora Global Database, DR secondary warm, quarterly switchover | 30 min / 5 min (typically seconds) |
| Idempotency keys, sessions | Supporting | DynamoDB global table (MREC), home Region per key | Minutes / seconds |
| Notification preferences | Supporting | DynamoDB global table (MREC) | Minutes / seconds |
| Reporting warehouse | Important, not time-critical | Backup copy to DR Region | 24 h / 24 h |
| SQL Server legacy reconciliation DB | Important | RDS for SQL Server cross-Region replica, promotion runbook | 2 h / 15 min ⚠ VERIFY edition |
The idempotency table matters in a failover. If a customer retries a payment after the Region switch and the DR replica does not yet have the key that the primary recorded, the payment could be processed twice. The design therefore combines a short replication lag alarm with a reconciliation job that compares payments by end_to_end_id after any failover. The idempotency lessons in Module 4 explain the keys themselves.
Common mistakes
- Treating the DR cluster as "free resilience" without an instance. A headless secondary cannot be promoted until you add an instance. Keep at least one instance running.
- Running a failover when a switchover would do. The flag
--allow-data-lossexists for a reason; use switchover whenever the primary is healthy. - Forgetting the application. The database promotion takes seconds to minutes; stale DNS and long-lived connection pools make the application recovery much longer. Test it.
- Using MREC global tables for balance updates. Last-writer-wins silently discards one of two concurrent debits. Keep each account's writes in one Region at a time, or use MRSC with conditional writes.
- Different parameter groups per Region. A promoted secondary does not inherit the primary's parameter group; settings such as
max_connectionscan differ and cause a slow brown-out after the switch. - No fail-back plan. Teams practise the failover and leave the system in the DR Region with nobody sure how to return. Include failback in every drill.
- Not checking engine versions. Managed switchover and failover need matching major and minor versions (and sometimes matching patch levels). Upgrade both sides together.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Switchover option is greyed out or the API rejects it | Primary and secondary engine versions are not compatible, or the secondary has no instance | Align versions (upgrade the secondary first for minor versions, per the documentation), add an instance |
| Application errors after promotion: "cannot execute INSERT in a read-only transaction" | Clients still resolve the old writer; connection pool not refreshed | Use the global writer endpoint, lower DNS TTL, restart or drain pools |
| Commits stall on the primary | rds.global_db_rpo is set and all secondaries are behind | Fix the lag (network, load) or raise the parameter; this is the intended safety behaviour |
AuroraGlobalDBRPOLag shows no data | Wrong Region or dimension; metric is read from the primary cluster | Query the primary Region with the primary's DBClusterIdentifier |
DynamoDB ReplicatedWriteConflictException | MRSC table: the same item is being written in two Regions at once | Retry with backoff; route each item's writes to one Region |
DynamoDB replica stuck in CREATING | Streams not enabled, or the table is not empty when creating MRSC | Enable streams (MREC); empty the table first (MRSC) |
| Promoted RDS replica missing encryption or alarms | Replica configuration is not inherited | Apply the DR configuration checklist; manage it with IaC |
AccessDenied on kms:CreateGrant during replica creation | KMS key policy in the target Region does not allow RDS | Add the required grants to the key policy ⚠ VERIFY exact actions |
Cost and cleanup
Paid. An Aurora global database needs at least one instance in each Region, plus storage, I/O and cross-Region replication charges. With two db.r6g.large instances the running cost is of the order of a few US dollars per hour ⚠ VERIFY current prices in the AWS Pricing Calculator (Module 1 lab), so deploy-and-destroy within an hour is possible, but the free alternative is the better path: do the runbook as a paper drill and run the C# advisor in a unit test with made-up lag values. A DynamoDB global table on on-demand billing costs pennies for a tiny table: replicated writes are billed in each Region ⚠ VERIFY.
Free alternative: a tabletop drill. Print the sequence diagram, assign roles (incident commander, database engineer, application engineer, scribe), and walk through the failover with a stopwatch and imaginary lag values.
Teardown, in this order (a global database cannot be deleted while it has secondary clusters):
# 1. Remove the secondary from the global cluster, then delete its instance and cluster.
aws rds remove-from-global-cluster --global-cluster-identifier hl-m12-ledger-global \
--db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:hl-m12-ledger-secondary \
--region us-east-1 --profile learn-sandbox
aws rds delete-db-instance --db-instance-identifier hl-m12-ledger-secondary-1 \
--region us-west-2 --profile learn-sandbox
aws rds delete-db-cluster --db-cluster-identifier hl-m12-ledger-secondary \
--skip-final-snapshot --region us-west-2 --profile learn-sandbox
# 2. Remove the primary from the global cluster, delete it, then delete the global cluster.
aws rds delete-db-instance --db-instance-identifier hl-m12-ledger-primary-1 \
--region us-east-1 --profile learn-sandbox
aws rds remove-from-global-cluster --global-cluster-identifier hl-m12-ledger-global \
--db-cluster-identifier arn:aws:rds:us-east-1:111122223333:cluster:hl-m12-ledger-primary \
--region us-east-1 --profile learn-sandbox
aws rds delete-db-cluster --db-cluster-identifier hl-m12-ledger-primary \
--skip-final-snapshot --region us-east-1 --profile learn-sandbox
aws rds delete-global-cluster --global-cluster-identifier hl-m12-ledger-global \
--region us-east-1 --profile learn-sandbox
# 3. Delete the DynamoDB global table (delete the replica first, wait 24 hours rule applies to some operations).
aws dynamodb update-table --table-name hl-m12-idempotency \
--replica-updates "Delete={RegionName=us-west-2}" --region us-east-1 --profile learn-sandbox
aws dynamodb delete-table --table-name hl-m12-idempotency --region us-east-1 --profile learn-sandbox
# 4. Confirm nothing tagged Project=cea-lab remains in either Region.
for r in us-east-1 us-west-2; do
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--region $r --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"
doneIf you used CDK or Terraform, run cdk destroy --all or terraform destroy instead. Remember that AWS notes that a table used as the source for a new replica cannot be deleted until 24 hours after the replica was created, so for a same-day lab plan to delete the replica table first or schedule the deletion for the next day ⚠ VERIFY the rule applies to your scenario.
Exercises
- 🟢 Guided: For each of Harbourline's five datastores in the table above, write the RTO and RPO and name the mechanism. Then walk through the sequence diagram for the ledger and write the one CLI command used at each step.
- 🟡 Challenge: Extend
FailoverAdvisorwith a rule: ifLagOfBestSecondaryis larger than the RPO and the primary has recovered partially (not healthy but responding), recommendWaitAndRetryfor up to 10 minutes. Write the xUnit v3 tests for all five outcomes without calling AWS. - 🔴 Live problem: At 14:05 the monitoring shows the us-east-1 ALB failing health checks, Aurora writer unreachable, and
AuroraGlobalDBRPOLaglast seen at 47 seconds six minutes ago. The AWS Health Dashboard shows a "service event" with no ETA. The business owner says "fail over now, I will take the data loss". Write the decision log: what you check, who approves, what you run, how you tell customers, and how you detect and reconcile lost payments afterwards.
Assignment
Produce a one-page database failover runbook for the Harbourline ledger and idempotency table: trigger criteria, approvers, numbered steps with exact commands, expected duration for each step, verification queries, communication points, and failback steps. Add the CDK or Terraform you would use to build it. Commit it to your portfolio repository under docs/runbooks/.
Solutions
Ledger: RTO 30 minutes, RPO 5 minutes, Aurora Global Database. Idempotency keys: RTO minutes, RPO seconds, DynamoDB global table (MREC). Preferences: same. Reporting warehouse: RTO and RPO 24 hours, cross-Region backup copy. Legacy SQL Server reconciliation: 2 hours and 15 minutes, RDS cross-Region replica.
Sequence commands: check lag with aws cloudwatch get-metric-statistics; stop writes by the application's read-only feature flag; promote with aws rds failover-global-cluster ... --allow-data-loss; verify the writer with SELECT pg_is_in_recovery(); which must return false; and later fail back with aws rds switchover-global-cluster. The point of the exercise is that every step has an owner and a command, so nobody discovers a missing permission during an outage.
Add a field PrimaryPartiallyResponding to FailoverFacts and a TimeSpan ImpairedFor. In Advise, after the health checks, add: if partially responding and ImpairedFor is less than 10 minutes, return WaitAndRetry. Tests: (1) no approvers, returns DoNotPromote; (2) healthy primary and matching versions, returns Switchover; (3) healthy primary and mismatched versions, returns DoNotPromote; (4) impaired primary, application healthy, returns WaitAndRetry; (5) impaired and application down, lag within RPO, returns FailoverAllowDataLoss with the reason string mentioning "within". A sixth test for lag above RPO checks the reason mentions "exceeds". The reason for pure functions is that each decision path becomes a one-line test and a reviewer can read the policy.
Check: is this a Regional event or only your stack (Health Dashboard, another service in the Region, a synthetic transaction from outside)? Lag of 47 seconds six minutes ago means the realistic loss is roughly the writes of the last six to seven minutes, above the 5-minute RPO, so the owner's explicit acceptance must be written down. Approvers: incident commander plus the business owner, recorded. Steps: put the app in read-only mode (stops new writes to the old writer), run the failover in the secondary Region, verify pg_is_in_recovery() is false and the application writes succeed, scale up the DR application tier, then switch traffic. Tell customers that payments initiated in a specific time window need to be re-checked. Afterwards restore the old primary's system snapshot (rds:unplanned-global-failover-...) into a scratch cluster, compare payments by end_to_end_id, and re-submit or reverse any payments the customer saw confirmed but the DR copy lacks. Keep the evidence for the post-incident review.
Knowledge check
Answer: A switchover is planned: Aurora waits for the secondary to be fully synchronised before swapping roles, so RPO is zero. A failover is unplanned: it does not wait, so recent writes that had not replicated can be lost, and you must explicitly allow data loss. Using a failover when the primary is healthy is wrong because it needlessly risks data.
Answer: Concurrent writes to the same item in two Regions are resolved by last writer wins, so one of two debits can disappear. Also, transactions are atomic only in their source Region. Mitigation: write each account from one home Region at a time, or use MRSC with conditional writes (accepting no transactions and higher latency).
Answer: It sets a maximum replication lag for Aurora PostgreSQL global databases. If every secondary is further behind than the limit, the primary blocks commits until at least one catches up. It trades write availability for a guaranteed RPO bound, and has a documented recommendation to leave it at the default in the secondary's parameter group for two-Region setups.
Answer: Repoint the application (there is no global writer endpoint), recreate replicas, rebuild the old primary as a replica later, and fix parameter groups and alarms. Promotion is one-way, so failback means building a new replica in the original Region and promoting it.
Answer: DynamoDB global tables in MRSC mode (synchronous replication to at least one other Region). MREC is asynchronous, so its RPO equals the replication delay. A backup copy has an RPO equal to the backup interval.
Interview questions
- Your CTO asks for "zero data loss across Regions" for a payments ledger. What do you ask back? What RTO they accept, the latency budget for writes, and which datastore. Zero RPO needs synchronous commit across Regions (for example DynamoDB MRSC), which costs write latency and features, or you accept seconds of loss with Aurora Global Database. Ask what the regulatory impact tolerance actually says.
- Explain split brain in a database failover. Two copies both accept writes because the old primary was not fenced. Aurora attempts write fencing but it is best-effort, so stop application writes first, use the global endpoint and low DNS TTL, and reconcile with the system snapshot.
- How would you prove to an auditor that DR works? Scheduled switchovers or restore tests with recorded timings, approvers, and results, plus alarms showing lag stayed within the RPO.
- When would you choose DynamoDB global tables over Aurora Global Database? Key-value access patterns that tolerate eventual consistency, multi-Region reads close to users, active-active writes with partitioned ownership; not a relational ledger.
Certification practice
A bank runs a PostgreSQL-compatible ledger on Aurora in us-east-1. Requirements: recover in another Region in under 30 minutes after a Regional failure, lose no more than a few seconds of data, and keep the application connection string unchanged after recovery. Which design fits best?
A. Aurora automated backups copied to the DR Region and restored on demand. B. Aurora Global Database with a secondary cluster in the DR Region and the global writer endpoint. C. An Aurora Multi-AZ cluster with three readers in us-east-1. D. A DynamoDB global table with MREC.
Answer: B. Global Database replicates continuously with seconds of lag, promotes a secondary in minutes, and the global writer endpoint keeps the connection string stable. A has an RPO equal to the backup interval and restores in hours. C protects against an AZ failure, not a Region. D is a different database model and would need a rewrite.
During a planned DR exercise, an operator must move the Aurora global database's writer to the secondary Region with no data loss, and the primary Region is fully healthy. Which action is correct?
A. failover-global-cluster --allow-data-loss
B. switchover-global-cluster
C. Detach the secondary cluster from the global database
D. Promote a reader in the primary Region
Answer: B. Switchover synchronises the secondary first, so RPO is zero. A risks data loss needlessly. C creates a standalone cluster and breaks the global relationship (the manual-failover technique). D does not change the Region.
An application writes the same DynamoDB items from two Regions. It needs strongly consistent reads of the latest value in both Regions and zero RPO, and can accept higher write latency. What do you choose?
A. A global table in MREC mode. B. A global table in MRSC mode across three Regions (or two plus a witness). C. Two independent tables with application-level replication. D. A global table in MREC mode with DAX in each Region.
Answer: B. MRSC replicates synchronously and gives strongly consistent reads of the latest data. A has a non-zero RPO and region-local strong reads. C reinvents replication with more failure modes. D adds a cache that can be stale because writes bypass DAX.
Completion checklist
- I can explain switchover versus failover in two sentences and say which one needs
--allow-data-loss - I can choose between Aurora Global Database, DynamoDB MREC, MRSC and a read replica for a stated RPO
- I completed the guided exercise and the failover runbook assignment
- I ran the cleanup steps (or did the paper drill) and confirmed nothing is still billing
Key takeaways
- Cross-Region recovery has a data half (asynchronous replication, so RPO greater than zero) and a control half (a rehearsed promotion and application redirect).
- Use switchover for planned moves and drills (zero loss), and failover only for real disasters, with the data loss accepted and recorded.
- DynamoDB MREC is last-writer-wins; MRSC gives zero RPO at the cost of latency and transactions. Neither makes a double-write-safe ledger on its own.
- Replicate everything around the database too: keys, secrets, parameter groups, alarms, application stack.
- A failover you have never timed is a guess. Record detect, decide, promote, repoint and verify times.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab: Resilient Account API (resilience pipeline, multi-AZ scaling and a DR plan)
Prove a .NET balance API survives a flaky dependency with a fault-injecting test backend, optionally deploy a three-AZ ECS service with target-tracking scaling and drill an AZ failure, then size a warm-standby DR plan for an RTO of 30 minutes and an RPO of 5 minutes.
AWS Backup in depth
Build a backup plan with vaults, Vault Lock, cross-Region and cross-account copies, and automated restore testing with validation, then produce restore evidence a bank can show an auditor.