Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Database failover across Regions

Choose and rehearse cross-Region database recovery: Aurora Global Database switchover and failover, DynamoDB global tables (MREC and MRSC), and RDS cross-Region read replica promotion, with a runbook and C# helpers that make the decision measurable.

advanced⏱ 2 h 30 min🔴 Paid — optionalSAA-C03SOA-C03SAP-C03DOP-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.RDS, AWSSDK.CloudWatch) · AWS CLI v2 · Aurora PostgreSQL 16/17 · DynamoDB global tables 2019.11.21 · Regions us-east-1 (primary) and us-west-2 (DR). Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the difference between an Aurora Global Database switchover (zero data loss, planned) and a failover (unplanned, possible data loss), and pick the right one for a given incident.
  • Compare DynamoDB global tables in MREC (multi-Region eventual consistency) and MRSC (multi-Region strong consistency) modes against an RPO target.
  • Promote an RDS cross-Region read replica and list the extra steps that Aurora Global Database does for you.
  • Write a database failover runbook with decision points, evidence capture and failback.
  • Use C# to read replication lag from CloudWatch and encode the "switchover or failover?" decision as a tested function.

Prerequisites

Why this matters in banking

Harbourline's ledger database holds the bookings behind every balance. The operational-resilience plan says the payments ledger must be back within 30 minutes (RTO) and may lose at most 5 minutes of bookings (RPO) if a whole Region is lost. The engineers have Multi-AZ and nightly backups, so they believe they are covered.

They are not. Multi-AZ protects against a lost Availability Zone, not a lost Region. Restoring last night's backup in another Region means up to a day of lost payments and hours of restore time. To meet 30 minutes and 5 minutes, there has to be a database already running in the second Region, kept current by replication, and a rehearsed procedure to make it the writer. This lesson is that procedure, and the honest discussion of what you can lose while you do it.

Regulators do not mandate a specific technology. They expect you to set an impact tolerance for each important business service and to show evidence that you can stay within it. Replication settings and a timed failover drill are that evidence. Always involve your risk and compliance teams; this lesson does not claim any compliance outcome.

The idea in plain language

Cross-Region recovery has two halves:

  1. Data: a second copy of the database in another Region that is only seconds behind.
  2. Control: a decision and a mechanism to make that copy the one that accepts writes, and to send the application to it.

Replication between Regions is asynchronous for most AWS databases. The primary commits a write and tells the client "done" before the remote copy has it. This is what keeps writes fast, and it is also exactly why the RPO is greater than zero: the writes in flight when the Region dies may never arrive.

Analogy: Harbourline's head office phones every booking through to the disaster-recovery branch. Normally a few seconds pass between the booking and the call. If the head office burns down mid-call, the DR branch has everything up to the last completed call. A switchover is closing head office on purpose after the last call has finished, so nothing is lost. A failover is the fire: you take what the DR branch has.

What problem each option solves

OptionReplicationTypical RPOTypical RTOWritersKey limits
Aurora Global DatabaseStorage-level, asynchronous, dedicated infrastructureSeconds (lag dependent)Minutes for failover; switchover typically under 30 s on current engine versionsOne primary Region; secondaries are read-onlySame major and minor engine versions needed for managed operations ⚠ VERIFY patch-level rules
DynamoDB global tables, MRECAsynchronous, per-itemReplication delay, usually about a secondSeconds to minutes (application redirect)All Regions accept writesLast-writer-wins conflicts; transactions atomic only in the source Region
DynamoDB global tables, MRSCSynchronous to at least one other Region before the write returnsZeroSeconds to minutes (application redirect)All replicas accept writesExactly three Regions (or two plus a witness); no transactions, no TTL, no LSIs; higher write latency
RDS cross-Region read replicaEngine-native, asynchronousReplica lag (seconds to minutes)Minutes: promote, then repointOne primary; replica is read-only until promotedPromotion is one-way: the replica leaves the replication relationship
AWS Backup cross-Region copyScheduled copies of backupsHours (backup interval)Hours (restore)None until restoredCheapest; see AWS Backup in depth

The figures are guidance from the AWS documentation. Your own measured numbers matter more, which is why the drill later in the lesson records them.

Aurora Global Database: switchover versus failover

An Aurora global database has one primary cluster and up to several secondary clusters in other Regions. The secondaries are read-only until promoted. Aurora provides two promotion operations:

  • Switchover (previously called "managed planned failover"). Use it when everything is healthy: Region rotation drills, planned maintenance, or failing back after an incident. Aurora waits for the chosen secondary to catch up, makes the old primary read-only, then promotes the secondary. RPO is zero. On current engine versions the read/write interruption is typically under 30 seconds ⚠ VERIFY your engine version qualifies.
  • Failover. Use it only for an unplanned Regional outage. It does not wait for replication to catch up, so you can lose the writes that had not yet arrived. In the CLI you must pass --allow-data-loss to make that explicit. Aurora attempts to snapshot the old primary's storage at the point of failure so that you can later recover missing rows, and attempts to fence writes on the old primary, but fencing is best-effort, which is why split brain (two clusters accepting writes) is a documented risk.

The recommended failover method is the managed one: when the old primary Region comes back, Aurora adds it back as a secondary, restoring the original topology. The alternative manual failover (detach the secondary, which becomes standalone, then build a new global database around it) exists for cases where managed failover is not possible, for example incompatible engine versions.

Regulators and auditors like the switchover for one more reason: the Aurora documentation describes regional rotation (moving the writer to another Region for a period) as a way to prove that disaster-recovery procedures really work. Harbourline's plan uses a quarterly switchover for exactly that.

Controlling RPO (Aurora PostgreSQL). The rds.global_db_rpo cluster parameter lets you declare an RPO in seconds (minimum 20). Aurora then blocks commits on the primary whenever every secondary is further behind than the limit. That turns RPO from a hope into a guarantee, at the price of write stalls when replication is slow. The documentation specifically recommends keeping the parameter at its default in the secondary's parameter group in a two-Region setup, otherwise a failover could pause writes on the newly promoted cluster. Treat it as a deliberate decision with your product owner, not a default.

Application endpoints. Use the global writer endpoint so that the application connection string does not change after a promotion. Keep DNS caching short (the documentation suggests a TTL as low as 5 seconds) and confirm that your pooled connections are recycled, otherwise some instances keep talking to the old writer.

DynamoDB global tables: MREC versus MRSC

A global table is a set of replica tables, one per Region, sharing a name, key schema and data. Version 2019.11.21 is the current version. There are two consistency modes, chosen when you create the table and not changeable afterwards:

  • MREC (default). Every replica accepts writes; changes replicate asynchronously, usually within a second. If the same item is written in two Regions at nearly the same time, last writer wins by timestamp. Strongly consistent reads are only strong with respect to writes in the same Region. Transactions (TransactWriteItems) are atomic only in the Region where they run, so another Region can briefly see half of a transaction.
  • MRSC. A write is replicated synchronously to at least one other Region before it returns, so RPO is zero and strongly consistent reads return the latest data in any replica. It must have exactly three Regions (three replicas, or two replicas and a witness), does not support transactions, TTL or local secondary indexes, and conflicting concurrent writes to one item in different Regions fail with ReplicatedWriteConflictException, which you retry. MRSC is only available in a documented list of Regions ⚠ VERIFY the list for your chosen pair.

The headline for a bank: MREC's last-writer-wins resolution and region-local transactions make it unsafe for a ledger that must apply debits and credits exactly once, unless the application is designed so each account is only ever written in one Region at a time (a "home Region per account"). MRSC removes the lost-update problem but not the application work, since you lose transactions. See DynamoDB banking correctness for how to design single-item conditional writes that survive these constraints. That is why the earlier lesson on active-active designs concluded that most ledgers stay single-writer, and why global tables fit better for session data, reference data, preferences and idempotency records.

RDS cross-Region read replicas

For RDS for PostgreSQL, MySQL, MariaDB (and SQL Server Enterprise ⚠ VERIFY edition and version rules), you can create a read replica in another Region. It is asynchronous, uses the engine's native replication, and can lag by seconds to minutes under load. To recover you promote the replica to a standalone instance, then do everything Aurora would do for you: repoint the application, fix parameter groups, create replicas of the new primary, and rebuild the old primary as a replica after the Region returns. Promotion is not reversible, and there is no managed switchback: failback means building a new replica in the original Region and promoting that.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure SQL Database active geo-replication and failover groupsAurora Global Database (failover and switchover)Aurora's replication runs at the storage layer; there is a dedicated global writer endpoint, like a failover group listener
SQL Server Always On availability groups across data centresRDS cross-Region read replica promotionRDS promotion is a one-way operation; you rebuild the replica afterwards
Planned vs forced failover in failover groups (allow data loss)Switchover vs failover with --allow-data-lossThe same trade-off, made explicit in the CLI flag
Cosmos DB multi-region writesDynamoDB global tables (MREC)Last-writer-wins by default; there is no custom conflict policy
Cosmos DB strong consistency, multi-region (not generally available for multi-write)DynamoDB MRSCSynchronous cross-Region commit at the cost of latency and transactions
Azure Traffic Manager health probes redirecting clientsRoute 53 failover records plus application logicThe database role change and the traffic change are two separate steps you must coordinate

Architecture

The sequence of a failover (unplanned) differs from a switchover (planned):

Security-boundary view: the failover action is a powerful, rare, high-impact permission. Put it behind a dedicated break-glass role in the management plane, require MFA, send CloudTrail events to a separate log account, and keep the DR Region's KMS keys, security groups and secrets replicated before the disaster. Secrets Manager secrets and KMS multi-Region keys need to be replicated ahead of time because you cannot create them in a hurry during an outage in the home Region.

Step-by-step implementation

This is a design and drill sequence. Run steps 1 to 3 in a sandbox only if you accept the cost shown later; otherwise do them as a paper exercise.

Set the targets and choose the mechanism

Write down the RTO and RPO per data store. For the Harbourline ledger: RTO 30 minutes, RPO 5 minutes. Aurora Global Database (seconds of lag, failover in minutes) meets that. For the idempotency-key table, DynamoDB MREC with a single home Region per key meets it cheaply.

Build the global database

Create the primary cluster, a global cluster that wraps it, and a secondary cluster in the DR Region with at least one instance. A secondary cluster with no instance (headless) must be given an instance before it can be promoted. Use the IaC below.

Replicate the things that are not in the database

Replicate secrets (Secrets Manager replica secrets), use multi-Region KMS keys or a separate key per Region, copy parameter groups and alarms, and make sure the DR Region has subnets, security groups and an application stack ready (a warm standby or pilot light).

Alarm on replication lag

Create a CloudWatch alarm on AuroraGlobalDBRPOLag for the secondary, with a threshold at half your RPO. Replication lag is the RPO you would actually get today.

Rehearse a switchover, then a failover

Do the switchover first: it is safe and it measures your real application recovery time. Rehearse the failover as a paper exercise with a stopwatch, or in a throwaway environment. Record times for detect, decide, promote, repoint, verify.

Fail back

Failback is a second change, with its own risk. After a failover, wait until Aurora has rebuilt the old primary as a secondary, check lag, then run a switchover during a quiet period.

AWS Console instructions

  • Aurora Global Database: Console → RDS → Databases → select the cluster → Actions → Add AWS Region to create a secondary. To promote, select the global database → Actions → Switch over or fail over global database and choose Switchover or Failover (allow data loss). Failover asks you to type confirm.
  • Replication lag: Console → CloudWatch → Metrics → RDS → Per-Cluster Metrics → AuroraGlobalDBRPOLag (milliseconds). Check it from the current primary's view.
  • DynamoDB global table: Console → DynamoDB → Tables → your table → Global tables tab → Create replica and pick the Region. For MRSC choose the consistency mode while creating the table's replicas ⚠ VERIFY exact console labels.
  • RDS read replica: Console → RDS → Databases → select the instance → Actions → Create read replica, and choose the destination Region. To promote: select the replica → Actions → Promote.

AWS CLI

# Check how far behind each secondary is (run against the primary's Region).
# AuroraGlobalDBRPOLag is in milliseconds.
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS --metric-name AuroraGlobalDBRPOLag \
  --dimensions Name=DBClusterIdentifier,Value=hl-m12-ledger-primary \
  --start-time "$(date -u -d '15 minutes ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 60 --statistics Maximum \
  --region us-east-1 --profile learn-sandbox

# PLANNED: zero data loss. Run with the Region of the CURRENT primary.
aws rds switchover-global-cluster \
  --global-cluster-identifier hl-m12-ledger-global \
  --target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:hl-m12-ledger-secondary \
  --region us-east-1 --profile learn-sandbox

# UNPLANNED: may lose recent writes. Run with the Region of the SECONDARY you are promoting.
aws rds failover-global-cluster \
  --global-cluster-identifier hl-m12-ledger-global \
  --target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:hl-m12-ledger-secondary \
  --allow-data-loss \
  --region us-west-2 --profile learn-sandbox

# Look for the system snapshot Aurora tries to take of the old primary after a failover.
aws rds describe-db-cluster-snapshots \
  --snapshot-type awsbackup --query "DBClusterSnapshots[?contains(DBClusterSnapshotIdentifier,'unplanned-global-failover')]" \
  --region us-west-2 --profile learn-sandbox

# DynamoDB: list replicas and their status for a global table.
aws dynamodb describe-table --table-name hl-m12-idempotency \
  --query "Table.Replicas[].{Region:RegionName,Status:ReplicaStatus}" \
  --region us-east-1 --profile learn-sandbox

# RDS (non-Aurora): promote a cross-Region read replica. One-way operation.
aws rds promote-read-replica --db-instance-identifier hl-m12-ledger-replica \
  --region us-west-2 --profile learn-sandbox

The account number 111122223333 is a placeholder. Note that the switchover is issued in the primary's Region and the failover in the secondary's Region, matching the AWS documentation. The describe-db-cluster-snapshots snapshot-type filter above is a best guess for the system snapshot ⚠ VERIFY, so if it returns nothing, list all snapshots for the cluster and look for the rds:unplanned-global-failover- prefix.

Infrastructure as Code

A global database spans two Regions, so it uses two stacks. The primary stack uses the high-level DatabaseCluster and wraps it in a CfnGlobalCluster; the secondary stack uses the low-level CfnDBCluster with the same global identifier. Cross-Region encryption needs a KMS key in each Region.

infra/cdk/LedgerGlobalDatabase.cs
using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.RDS;
using Constructs;

namespace Harbourline.Ha.Infra;

public sealed class LedgerPrimaryStack : Stack
{
    public const string GlobalId = "hl-m12-ledger-global";
    public DatabaseCluster Cluster { get; }

    public LedgerPrimaryStack(Construct scope, string id, IVpc vpc, IStackProps props)
        : base(scope, id, props)
    {
        var key = new Key(this, "DbKey", new KeyProps { EnableKeyRotation = true });

        Cluster = new DatabaseCluster(this, "Primary", new DatabaseClusterProps
        {
            ClusterIdentifier = "hl-m12-ledger-primary",
            Engine = DatabaseClusterEngine.AuroraPostgres(new AuroraPostgresClusterEngineProps
            {
                // Pick a version that supports global databases in both Regions. ⚠ VERIFY
                Version = AuroraPostgresEngineVersion.VER_16_8
            }),
            Vpc = vpc,
            VpcSubnets = new SubnetSelection { SubnetType = SubnetType.PRIVATE_ISOLATED },
            StorageEncrypted = true,
            StorageEncryptionKey = key,
            Writer = ClusterInstance.Provisioned("writer", new ProvisionedClusterInstanceProps
            {
                InstanceType = Amazon.CDK.AWS.EC2.InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE)
            }),
            Readers = new[]
            {
                ClusterInstance.Provisioned("reader", new ProvisionedClusterInstanceProps
                {
                    InstanceType = Amazon.CDK.AWS.EC2.InstanceType.Of(InstanceClass.R6G, InstanceSize.LARGE)
                })
            },
            Backup = new BackupProps { Retention = Duration.Days(14) },
            DeletionProtection = false, // learning shortcut: see Security considerations
            RemovalPolicy = RemovalPolicy.DESTROY
        });

        // Wrap the existing primary cluster in a global cluster.
        _ = new CfnGlobalCluster(this, "Global", new CfnGlobalClusterProps
        {
            GlobalClusterIdentifier = GlobalId,
            SourceDbClusterIdentifier = Cluster.ClusterArn
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m12-database-failover");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
    }
}

public sealed class LedgerSecondaryStack : Stack
{
    public LedgerSecondaryStack(Construct scope, string id, string subnetGroupName,
        string securityGroupId, IStackProps props)
        : base(scope, id, props)
    {
        // A separate key in the DR Region. Replicated storage is re-encrypted with it.
        var key = new Key(this, "DbKey", new KeyProps { EnableKeyRotation = true });

        var secondary = new CfnDBCluster(this, "Secondary", new CfnDBClusterProps
        {
            DbClusterIdentifier = "hl-m12-ledger-secondary",
            GlobalClusterIdentifier = LedgerPrimaryStack.GlobalId,
            Engine = "aurora-postgresql",
            EngineVersion = "16.8", // must match the primary's major and minor version
            DbSubnetGroupName = subnetGroupName,
            VpcSecurityGroupIds = new[] { securityGroupId },
            StorageEncrypted = true,
            KmsKeyId = key.KeyArn
            // No MasterUsername or password: a secondary inherits data from the primary.
        });

        // A secondary needs at least one instance before it can be promoted.
        _ = new CfnDBInstance(this, "SecondaryInstance", new CfnDBInstanceProps
        {
            DbClusterIdentifier = secondary.Ref,
            DbInstanceClass = "db.r6g.large",
            Engine = "aurora-postgresql"
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m12-database-failover");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
    }
}

DynamoDB global table with the L2 TableV2 construct, which creates a global table with replicas:

infra/cdk/IdempotencyGlobalTable.cs
using Amazon.CDK;
using Amazon.CDK.AWS.DynamoDB;
using Constructs;

namespace Harbourline.Ha.Infra;

public sealed class IdempotencyTableStack : Stack
{
    public IdempotencyTableStack(Construct scope, string id, IStackProps props)
        : base(scope, id, props)
    {
        _ = new TableV2(this, "Idempotency", new TablePropsV2
        {
            TableName = "hl-m12-idempotency",
            PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
            Billing = Billing.OnDemand(),
            TimeToLiveAttribute = "expiresAt",
            PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification
            {
                PointInTimeRecoveryEnabled = true
            },
            // Replicas in other Regions. The stack's own Region is the first replica.
            Replicas = new[] { new ReplicaTableProps { Region = "us-west-2" } },
            RemovalPolicy = RemovalPolicy.DESTROY
        });
        // This is MREC (the default). MRSC needs three Regions and a consistency
        // setting at creation. ⚠ VERIFY the TablePropsV2 property name in your CDK version.
    }
}

Complete code

Two small, testable pieces. First, a pure function that turns the facts you can measure into a recommendation, so the logic is reviewed and unit-tested rather than improvised at 03:00.

src/Harbourline.Ha.Failover/FailoverAdvisor.cs
namespace Harbourline.Ha.Failover;

public enum Recommendation
{
    Switchover,      // planned, zero data loss
    FailoverAllowDataLoss,
    WaitAndRetry,
    DoNotPromote
}

public sealed record FailoverFacts(
    bool PrimaryRegionHealthy,
    bool ApplicationHealthy,
    TimeSpan LagOfBestSecondary,
    TimeSpan RpoTarget,
    bool SecondaryEngineVersionMatchesPrimary,
    bool TwoApproversRecorded);

public static class FailoverAdvisor
{
    public static (Recommendation Action, string Reason) Advise(FailoverFacts f)
    {
        if (!f.TwoApproversRecorded)
            return (Recommendation.DoNotPromote, "A disaster declaration needs two recorded approvers.");

        if (f.PrimaryRegionHealthy)
        {
            // Everything works: only a planned switchover is appropriate.
            return f.SecondaryEngineVersionMatchesPrimary
                ? (Recommendation.Switchover, "Primary is healthy; use switchover for zero data loss.")
                : (Recommendation.DoNotPromote, "Engine versions differ; managed operations need matching versions.");
        }

        // Primary Region is impaired from here on.
        if (f.ApplicationHealthy)
            return (Recommendation.WaitAndRetry, "Application still serves traffic; keep watching before declaring.");

        return f.LagOfBestSecondary <= f.RpoTarget
            ? (Recommendation.FailoverAllowDataLoss,
               $"Best secondary lag {f.LagOfBestSecondary.TotalSeconds:F0}s is within the {f.RpoTarget.TotalSeconds:F0}s RPO.")
            : (Recommendation.FailoverAllowDataLoss,
               $"Lag {f.LagOfBestSecondary.TotalSeconds:F0}s exceeds the RPO: the business must accept the data loss explicitly.");
    }
}

Second, a console tool that reads real lag from CloudWatch and executes the AWS call only when you pass --execute.

src/Harbourline.Ha.Failover/Program.cs
using Amazon;
using Amazon.CloudWatch;
using Amazon.CloudWatch.Model;
using Amazon.RDS;
using Amazon.RDS.Model;
using Harbourline.Ha.Failover;

var globalId = args.ElementAtOrDefault(0) ?? "hl-m12-ledger-global";
var primaryClusterId = args.ElementAtOrDefault(1) ?? "hl-m12-ledger-primary";
var targetArn = args.ElementAtOrDefault(2)
    ?? throw new ArgumentException("Usage: <globalId> <primaryClusterId> <targetClusterArn> [--execute]");
var execute = args.Contains("--execute");

using var cw = new AmazonCloudWatchClient(RegionEndpoint.USEast1);

// 1. Read the worst lag over the last 5 minutes (milliseconds).
var stats = await cw.GetMetricStatisticsAsync(new GetMetricStatisticsRequest
{
    Namespace = "AWS/RDS",
    MetricName = "AuroraGlobalDBRPOLag",
    Dimensions = [new Dimension { Name = "DBClusterIdentifier", Value = primaryClusterId }],
    StartTime = DateTime.UtcNow.AddMinutes(-5),
    EndTime = DateTime.UtcNow,
    Period = 60,
    Statistics = ["Maximum"]
});

var lagMs = stats.Datapoints.Count == 0 ? double.NaN : stats.Datapoints.Max(d => d.Maximum ?? 0);
Console.WriteLine($"Worst replication lag (5 min): {lagMs:F0} ms");

var facts = new FailoverFacts(
    PrimaryRegionHealthy: args.Contains("--primary-healthy"),
    ApplicationHealthy: args.Contains("--app-healthy"),
    LagOfBestSecondary: double.IsNaN(lagMs) ? TimeSpan.MaxValue : TimeSpan.FromMilliseconds(lagMs),
    RpoTarget: TimeSpan.FromMinutes(5),
    SecondaryEngineVersionMatchesPrimary: true,
    TwoApproversRecorded: args.Contains("--two-approvers"));

var (action, reason) = FailoverAdvisor.Advise(facts);
Console.WriteLine($"Advice: {action}. {reason}");

if (!execute || action is Recommendation.WaitAndRetry or Recommendation.DoNotPromote)
{
    Console.WriteLine("Dry run: nothing changed. Pass --execute to act on the advice.");
    return;
}

// 2. Switchover is issued in the primary's Region, failover in the secondary's Region.
//    The target ARN tells us the secondary's Region.
var targetRegion = RegionEndpoint.GetBySystemName(targetArn.Split(':')[3]);
using var rds = new AmazonRDSClient(action == Recommendation.Switchover ? RegionEndpoint.USEast1 : targetRegion);

if (action == Recommendation.Switchover)
{
    await rds.SwitchoverGlobalClusterAsync(new SwitchoverGlobalClusterRequest
    {
        GlobalClusterIdentifier = globalId,
        TargetDbClusterIdentifier = targetArn
    });
}
else
{
    await rds.FailoverGlobalClusterAsync(new FailoverGlobalClusterRequest
    {
        GlobalClusterIdentifier = globalId,
        TargetDbClusterIdentifier = targetArn,
        AllowDataLoss = true
    });
}

Console.WriteLine($"Requested {action} at {DateTime.UtcNow:O}. Record this timestamp in the drill log.");

Package references: AWSSDK.RDS and AWSSDK.CloudWatch 4.x (look up the exact versions on NuGet). The request property names (TargetDbClusterIdentifier, AllowDataLoss) follow the SDK's PascalCase mapping of the API fields ⚠ VERIFY against the SDK version you install.

Code explained line by line

  1. FailoverFacts collects the observable inputs (health, lag, approvals). Making them a record means a test can construct any scenario without AWS.
  2. Advise checks approvals first: a failover is a business decision, so the code refuses to recommend promotion without two recorded approvers.
  3. If the primary Region is healthy, the only legitimate operation is a switchover, which waits for replication to finish. This prevents someone running the data-loss path during a drill.
  4. If the primary is impaired but the application still serves traffic, the advisor says wait. Declaring a disaster too early is itself a risk.
  5. When lag exceeds the RPO, the advisor still returns the failover but with a reason that spells out that data loss exceeds target, so the person approving cannot miss it.
  6. In Program.cs, GetMetricStatisticsAsync reads AuroraGlobalDBRPOLag as Maximum over 5 minutes. The maximum is used because the worst lag in the window is the realistic data-loss figure.
  7. The tool is dry run by default. Requiring --execute makes accidental failovers during a demo much less likely.
  8. The RDS client is created in the primary's Region for a switchover and in the target's Region for a failover, mirroring the AWS guidance about which Region to call.
  9. The last line prints a timestamp: your RTO measurement starts from the moment the decision was acted on.

Security considerations

⚠ Learning shortcut — not for production

The IaC above sets DeletionProtection = false, skip_final_snapshot = true and RemovalPolicy.DESTROY so you can tear down the lab. In production, keep deletion protection on, take final snapshots, keep backups in a Vault Lock vault in a separate account, and use manage_master_user_password (Secrets Manager) with replicated secrets. Never put a password in IaC variables.

  • Least privilege for the failover action. rds:FailoverGlobalCluster and rds:SwitchoverGlobalCluster belong to a break-glass role, assumed with MFA, with session tags recording the incident number. Everyone else gets read-only.
  • Encryption. Each Region needs a usable KMS key. A global database encrypts storage in each Region with a key in that Region. Verify that the DR Region's key policy allows the application role and the RDS service.
  • Secrets and configuration. Replicate Secrets Manager secrets and Parameter Store values to the DR Region ahead of time. A failover that cannot read the database password is not a failover.
  • Data residency. Replicating a UK customer ledger to a US Region is a transfer of personal data. For Harbourline's real design, the DR Region for UK data would be another UK or EEA Region. us-west-2 is used here for lab convenience only. Ask your data-protection officer; see UK GDPR and data residency.
  • Split brain. Two clusters accepting writes means two versions of the truth, which for a bank is a reconciliation incident. Stop application writes before the promotion, use the global writer endpoint and short DNS caching, and keep the old primary's system snapshot so you can reconcile.
  • Evidence. CloudTrail logs the API calls. Store the drill log (timestamps, approvers, lag, results) with your operational-resilience evidence.

Banking use case

Harbourline's resilience plan assigns each datastore a tier:

DatastoreTierMechanismRTO / RPO
Ledger (PostgreSQL)CriticalAurora Global Database, DR secondary warm, quarterly switchover30 min / 5 min (typically seconds)
Idempotency keys, sessionsSupportingDynamoDB global table (MREC), home Region per keyMinutes / seconds
Notification preferencesSupportingDynamoDB global table (MREC)Minutes / seconds
Reporting warehouseImportant, not time-criticalBackup copy to DR Region24 h / 24 h
SQL Server legacy reconciliation DBImportantRDS for SQL Server cross-Region replica, promotion runbook2 h / 15 min ⚠ VERIFY edition

The idempotency table matters in a failover. If a customer retries a payment after the Region switch and the DR replica does not yet have the key that the primary recorded, the payment could be processed twice. The design therefore combines a short replication lag alarm with a reconciliation job that compares payments by end_to_end_id after any failover. The idempotency lessons in Module 4 explain the keys themselves.

Common mistakes

  • Treating the DR cluster as "free resilience" without an instance. A headless secondary cannot be promoted until you add an instance. Keep at least one instance running.
  • Running a failover when a switchover would do. The flag --allow-data-loss exists for a reason; use switchover whenever the primary is healthy.
  • Forgetting the application. The database promotion takes seconds to minutes; stale DNS and long-lived connection pools make the application recovery much longer. Test it.
  • Using MREC global tables for balance updates. Last-writer-wins silently discards one of two concurrent debits. Keep each account's writes in one Region at a time, or use MRSC with conditional writes.
  • Different parameter groups per Region. A promoted secondary does not inherit the primary's parameter group; settings such as max_connections can differ and cause a slow brown-out after the switch.
  • No fail-back plan. Teams practise the failover and leave the system in the DR Region with nobody sure how to return. Include failback in every drill.
  • Not checking engine versions. Managed switchover and failover need matching major and minor versions (and sometimes matching patch levels). Upgrade both sides together.

Troubleshooting

SymptomLikely causeFix
Switchover option is greyed out or the API rejects itPrimary and secondary engine versions are not compatible, or the secondary has no instanceAlign versions (upgrade the secondary first for minor versions, per the documentation), add an instance
Application errors after promotion: "cannot execute INSERT in a read-only transaction"Clients still resolve the old writer; connection pool not refreshedUse the global writer endpoint, lower DNS TTL, restart or drain pools
Commits stall on the primaryrds.global_db_rpo is set and all secondaries are behindFix the lag (network, load) or raise the parameter; this is the intended safety behaviour
AuroraGlobalDBRPOLag shows no dataWrong Region or dimension; metric is read from the primary clusterQuery the primary Region with the primary's DBClusterIdentifier
DynamoDB ReplicatedWriteConflictExceptionMRSC table: the same item is being written in two Regions at onceRetry with backoff; route each item's writes to one Region
DynamoDB replica stuck in CREATINGStreams not enabled, or the table is not empty when creating MRSCEnable streams (MREC); empty the table first (MRSC)
Promoted RDS replica missing encryption or alarmsReplica configuration is not inheritedApply the DR configuration checklist; manage it with IaC
AccessDenied on kms:CreateGrant during replica creationKMS key policy in the target Region does not allow RDSAdd the required grants to the key policy ⚠ VERIFY exact actions

Cost and cleanup

Paid. An Aurora global database needs at least one instance in each Region, plus storage, I/O and cross-Region replication charges. With two db.r6g.large instances the running cost is of the order of a few US dollars per hour ⚠ VERIFY current prices in the AWS Pricing Calculator (Module 1 lab), so deploy-and-destroy within an hour is possible, but the free alternative is the better path: do the runbook as a paper drill and run the C# advisor in a unit test with made-up lag values. A DynamoDB global table on on-demand billing costs pennies for a tiny table: replicated writes are billed in each Region ⚠ VERIFY.

Free alternative: a tabletop drill. Print the sequence diagram, assign roles (incident commander, database engineer, application engineer, scribe), and walk through the failover with a stopwatch and imaginary lag values.

Teardown, in this order (a global database cannot be deleted while it has secondary clusters):

# 1. Remove the secondary from the global cluster, then delete its instance and cluster.
aws rds remove-from-global-cluster --global-cluster-identifier hl-m12-ledger-global \
  --db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:hl-m12-ledger-secondary \
  --region us-east-1 --profile learn-sandbox
aws rds delete-db-instance --db-instance-identifier hl-m12-ledger-secondary-1 \
  --region us-west-2 --profile learn-sandbox
aws rds delete-db-cluster --db-cluster-identifier hl-m12-ledger-secondary \
  --skip-final-snapshot --region us-west-2 --profile learn-sandbox

# 2. Remove the primary from the global cluster, delete it, then delete the global cluster.
aws rds delete-db-instance --db-instance-identifier hl-m12-ledger-primary-1 \
  --region us-east-1 --profile learn-sandbox
aws rds remove-from-global-cluster --global-cluster-identifier hl-m12-ledger-global \
  --db-cluster-identifier arn:aws:rds:us-east-1:111122223333:cluster:hl-m12-ledger-primary \
  --region us-east-1 --profile learn-sandbox
aws rds delete-db-cluster --db-cluster-identifier hl-m12-ledger-primary \
  --skip-final-snapshot --region us-east-1 --profile learn-sandbox
aws rds delete-global-cluster --global-cluster-identifier hl-m12-ledger-global \
  --region us-east-1 --profile learn-sandbox

# 3. Delete the DynamoDB global table (delete the replica first, wait 24 hours rule applies to some operations).
aws dynamodb update-table --table-name hl-m12-idempotency \
  --replica-updates "Delete={RegionName=us-west-2}" --region us-east-1 --profile learn-sandbox
aws dynamodb delete-table --table-name hl-m12-idempotency --region us-east-1 --profile learn-sandbox

# 4. Confirm nothing tagged Project=cea-lab remains in either Region.
for r in us-east-1 us-west-2; do
  aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
    --region $r --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"
done

If you used CDK or Terraform, run cdk destroy --all or terraform destroy instead. Remember that AWS notes that a table used as the source for a new replica cannot be deleted until 24 hours after the replica was created, so for a same-day lab plan to delete the replica table first or schedule the deletion for the next day ⚠ VERIFY the rule applies to your scenario.

Exercises

  • 🟢 Guided: For each of Harbourline's five datastores in the table above, write the RTO and RPO and name the mechanism. Then walk through the sequence diagram for the ledger and write the one CLI command used at each step.
  • 🟡 Challenge: Extend FailoverAdvisor with a rule: if LagOfBestSecondary is larger than the RPO and the primary has recovered partially (not healthy but responding), recommend WaitAndRetry for up to 10 minutes. Write the xUnit v3 tests for all five outcomes without calling AWS.
  • 🔴 Live problem: At 14:05 the monitoring shows the us-east-1 ALB failing health checks, Aurora writer unreachable, and AuroraGlobalDBRPOLag last seen at 47 seconds six minutes ago. The AWS Health Dashboard shows a "service event" with no ETA. The business owner says "fail over now, I will take the data loss". Write the decision log: what you check, who approves, what you run, how you tell customers, and how you detect and reconcile lost payments afterwards.

Assignment

Produce a one-page database failover runbook for the Harbourline ledger and idempotency table: trigger criteria, approvers, numbered steps with exact commands, expected duration for each step, verification queries, communication points, and failback steps. Add the CDK or Terraform you would use to build it. Commit it to your portfolio repository under docs/runbooks/.

Solutions

Knowledge check

Interview questions

  • Your CTO asks for "zero data loss across Regions" for a payments ledger. What do you ask back? What RTO they accept, the latency budget for writes, and which datastore. Zero RPO needs synchronous commit across Regions (for example DynamoDB MRSC), which costs write latency and features, or you accept seconds of loss with Aurora Global Database. Ask what the regulatory impact tolerance actually says.
  • Explain split brain in a database failover. Two copies both accept writes because the old primary was not fenced. Aurora attempts write fencing but it is best-effort, so stop application writes first, use the global endpoint and low DNS TTL, and reconcile with the system snapshot.
  • How would you prove to an auditor that DR works? Scheduled switchovers or restore tests with recorded timings, approvers, and results, plus alarms showing lag stayed within the RPO.
  • When would you choose DynamoDB global tables over Aurora Global Database? Key-value access patterns that tolerate eventual consistency, multi-Region reads close to users, active-active writes with partitioned ownership; not a relational ledger.

Certification practice

Completion checklist

  • I can explain switchover versus failover in two sentences and say which one needs --allow-data-loss
  • I can choose between Aurora Global Database, DynamoDB MREC, MRSC and a read replica for a stated RPO
  • I completed the guided exercise and the failover runbook assignment
  • I ran the cleanup steps (or did the paper drill) and confirmed nothing is still billing

Key takeaways

  • Cross-Region recovery has a data half (asynchronous replication, so RPO greater than zero) and a control half (a rehearsed promotion and application redirect).
  • Use switchover for planned moves and drills (zero loss), and failover only for real disasters, with the data loss accepted and recorded.
  • DynamoDB MREC is last-writer-wins; MRSC gives zero RPO at the cost of latency and transactions. Neither makes a double-write-safe ledger on its own.
  • Replicate everything around the database too: keys, secrets, parameter groups, alarms, application stack.
  • A failover you have never timed is a guess. Record detect, decide, promote, repoint and verify times.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page