Skip to Content
Cloud Engineering Academy
6. Databases & Data Architecture

Multi-AZ, read replicas, backups and point-in-time recovery

Know exactly what Multi-AZ instances, Multi-AZ DB clusters, read replicas, automated backups, snapshots and point-in-time restore protect against, map them to RPO and RTO, and run a repeatable, evidenced restore drill from C#.

intermediate⏱ 2 h 🟡 Low costSAA-C03DEA-C01DVA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.RDS 4.1.0.2) · AWS CLI v2 · RDS for PostgreSQL 17 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Say what each protection mechanism protects against: Multi-AZ DB instance, Multi-AZ DB cluster, read replicas, automated backups, manual snapshots, point-in-time restore (PITR), cross-Region automated backups and AWS Backup.
  • Translate business requirements (RPO and RTO) into a configuration, and explain why "we have Multi-AZ" is not a backup strategy.
  • Configure backup retention, windows and snapshots with the CLI and IaC, and know which changes cause an outage.
  • Run a restore drill that measures time to restore and proves the restored data is complete, using a C# orchestrator and the module project's fingerprint command.
  • Explain the RDS for SQL Server specifics: Always On Multi-AZ, Enterprise-only replicas, and per-database PITR consistency.

Prerequisites

Why this matters in banking

At 16:40 on a Friday, a Harbourline release runs a data fix against the ledger. It contains an UPDATE with a missing WHERE clause, and every savings account's interest rate becomes 0.00%. The database is Multi-AZ, so the standby faithfully applies the same mistake within milliseconds. The read replica copies it a second later.

What saves the bank is point-in-time restore to 16:39, onto a new instance, followed by a careful repair of the damaged rows from the restored copy. Except that nobody had ever practised it: the team discovers mid-incident that the restored instance came up with the default parameter group and the wrong security group, so the application can't reach it. Two hours pass.

Under operational resilience rules, Harbourline must be able to show that important business services stay within their impact tolerance, and backups that have never been restored are not evidence. This lesson separates the mechanisms that protect against infrastructure failure from those that protect against data damage, and turns restores into a routine drill.

The idea in plain language

There are two different kinds of bad day:

  1. Something breaks: a disk, a host, an Availability Zone, a Region. You want the database back quickly with no data lost. Replication handles this: a standby or replica that already has the data.
  2. Something is damaged: a bad deployment, a malicious insider, ransomware, an accidental DELETE. Replication makes this worse, because it copies the damage everywhere instantly. You need a copy from before the damage: backups, snapshots and point-in-time restore.

Two numbers describe what the business can tolerate:

  • RPO (recovery point objective): how much data you may lose, measured in time. "RPO 5 minutes" means at most the last 5 minutes of changes.
  • RTO (recovery time objective): how long the service may be down.

Analogy: Multi-AZ is a second teller standing next to the first, writing every entry into an identical ledger at the same moment. If the first teller faints, the second carries on. But if the first teller writes the wrong amount, so does the second. Backups are the photocopies of the ledger locked in the vault every night, plus the carbon copies of each day's slips. They are the only thing that can tell you what the ledger looked like before the mistake.

What problem each mechanism solves

MechanismProtects againstData loss (RPO)Downtime (RTO)Readable?Notes
Multi-AZ DB instance (one standby)Host, storage or AZ failure; also used during patching~0 (synchronous)Typically 60–120 s failoverNo, standby is not readableDNS name stays the same; clients reconnect
Multi-AZ DB cluster (writer + 2 readable standbys, PostgreSQL/MySQL)Same, plus read scaling~0 (semi-synchronous: one standby must acknowledge)Usually faster than Multi-AZ instance ⚠ VERIFY current figuresYesSpecific instance classes only (db.m6gd, db.r6gd, db.m8gd... with local NVMe)
Read replica (same or other Region)Read load; can be promoted for DRReplica lag (seconds+)Minutes (manual promotion)YesAsynchronous; up to 15 per primary; SQL Server: Enterprise only
Aurora replicasInstance or AZ failure; read load~0 (shared storage)Typically under 60 s, often under 30 sYesSee Aurora lesson
Automated backups + PITRData damage, deletion, corruptionUp to ~5 minutes (logs uploaded every 5 minutes)Minutes to hours (new instance + data loading)NoRetention 0–35 days; restore creates a new instance
Manual snapshotsLong-term restore points; before risky changesTime of snapshotMinutes to hoursNoDon't expire; 100 per Region quota; can be copied and shared
Cross-Region automated backupsLoss of the Region (for restores)Few minutesHours (restore in another Region)NoSnapshots and logs copied to a second Region
AWS Backup (with vault lock, cross-account copy)Ransomware, insider deletion, policy compliancePer planHoursNoCentral policies, immutable vaults, reporting

Key facts behind the table (checked 2026-11-18):

  • Retention can be 0–35 days for DB instances. 0 disables automated backups and PITR. The console default is 7 days; the API/CLI default is 1 day. Multi-AZ DB clusters need 1–35.
  • Changing retention between 0 and non-zero causes an outage. Changing between two non-zero values doesn't.
  • RDS uploads transaction logs to S3 every five minutes, so LatestRestorableTime is typically within about five minutes of now.
  • Restores always create a new instance. A restored instance gets the default parameter and option groups unless you specify others on the API/CLI. Console restores let you change them afterwards.
  • Free backup storage equals 100% of your provisioned database storage per Region. Beyond that you pay per GB-month.
  • Snapshots of an encrypted instance stay encrypted with the same key. You can choose a different key when copying a snapshot.
  • If you delete an instance without Retain automated backups, its automated backups go with it. Manual snapshots survive instance deletion.
  • SQL Server: PITR restores each database in the instance to within one second of each other, so transactions spanning databases may be inconsistent. Changing a database's recovery model (for example to BULK_LOGGED) can break the log chain. SQL Server read replicas don't support PITR on the replica itself.

Why Multi-AZ is not a backup

Multi-AZ replicates every change, including your mistakes, within the same commit. It protects availability, not integrity. Every bank database therefore needs both high availability and a tested restore path.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Always On AG synchronous secondary (no read)Multi-AZ DB instanceManaged; standby unreadable; you don't see the cluster
AG readable secondariesMulti-AZ DB cluster (PG/MySQL) or read replicasReadable standbys need specific instance classes
Full + differential + log backups, maintenance plansAutomated backups (daily snapshot + 5-minute log upload)No backup jobs to write; retention is a setting
RESTORE ... WITH STOPATPoint-in-time restore to a new instanceYou can't restore "over" the existing instance
BACKUP DATABASE ... COPY_ONLY before a releaseManual snapshotInstance-level, not per database
Azure SQL LTR (long-term retention)Manual snapshots / AWS Backup plans / snapshot export to S3AWS Backup adds vault lock and cross-account copy
Geo-restore from geo-redundant backupsCross-Region automated backupsYou enable and pay for it per instance
Azure Backup immutable vaultsAWS Backup Vault Lock (compliance mode)Same idea: nobody, including root, can delete early

Architecture

Security-boundary view.

  • Snapshots and automated backups are encrypted with the instance's KMS key. Copies to another account need a customer managed key shared with that account.
  • The backup account is a separate AWS account with its own IAM and a locked vault. A compromised production account can't delete those copies.
  • Restore permissions (rds:RestoreDBInstanceToPointInTime, rds:RestoreDBInstanceFromDBSnapshot) are powerful: whoever can restore can read the data. Grant them only to a break-glass or DBA role, and alert on their use with CloudTrail.

Mapping requirements to configuration

Service tier (fictional Harbourline policy)RPORTOConfiguration
Tier 1: ledger, payments≤ 5 min (data damage); ~0 (infrastructure)≤ 15 min (infrastructure); ≤ 4 h (data damage)Aurora or Multi-AZ; 35-day retention; cross-Region automated backups; AWS Backup copies to a locked vault in the backup account; monthly restore drill
Tier 2: customer servicing≤ 15 min≤ 4 hMulti-AZ instance; 14-day retention; AWS Backup daily; quarterly drill
Tier 3: internal tools≤ 24 h≤ 1 business daySingle-AZ; 7-day retention; annual drill

Step-by-step implementation

The drill below uses the lab instance from the RDS lesson, or the module project's hl-m06-ledger instance. If you prefer the free path, follow the local version at the end of this section.

Check the protection settings

Confirm the instance has backup retention of at least 1 (7+ in production), and note LatestRestorableTime. If retention is 0, there is nothing to restore from.

Take a manual snapshot before a "risky change"

Banks take a named snapshot before every schema migration or data fix. The snapshot doesn't expire, so delete it when the change is signed off.

Simulate damage

In a test schema, record the time, then run a deliberately bad UPDATE (on fictional data only). Note the exact UTC time just before the damage. This is your restore target.

Restore to a point in time, correctly

Restore to a new instance with the same subnet group, security groups and parameter group, not publicly accessible, with IAM authentication on, and with drill tags. The C# orchestrator in Complete code does exactly this and measures the time.

Verify the data, not just the instance

An "available" instance proves only that RDS worked. Run the ledger fingerprint command (in labs/m06-banking-data-store) against the restored instance. It prints counts, totals and a SHA-256 digest over every transaction. Compare it with the expected state, for example a fingerprint taken just before the damage. The module project's scripts/restore-drill.sh automates source-vs-restore comparison.

Record evidence and clean up

Keep the evidence file (restore duration, target time, fingerprint comparison). Delete the restored instance and the manual snapshot. File the evidence where your operational resilience team expects it.

Free local version: in labs/m06-banking-data-store, dotnet test includes RestoreFingerprintTests. One test backs up a seeded SQLite ledger with SQLite's online backup API, "restores" it into a fresh database and proves the fingerprints match. Another simulates a restore point taken before a posting and proves the fingerprint detects the missing transaction. The mechanics of measuring, verifying and recording are identical.

AWS Console instructions

  • Console → RDS → Databases → your instance → Maintenance & backups: see the retention period, backup window and latest restore time. To change retention: Modify → Backup → Backup retention period.
  • Manual snapshot: select the instance, then Actions → Take snapshot. Name it hl-m06-l5-pre-release-<date>.
  • PITR: Console → RDS → Automated backups → select the instance → Actions → Restore to point in time. Choose Custom and enter the time (shown in your local time zone; check the UTC offset). Then set the identifier, DB subnet group, Public access: No, the security group and the parameter group (under Additional configuration).
  • Cross-Region automated backups: Automated backups → Replicated tab → Manage cross-Region replication (or from the instance's Actions menu ⚠ VERIFY the current menu location).
  • Failover test (Multi-AZ): Actions → Reboot → Reboot with failover.

AWS CLI

PROFILE=learn-sandbox; REGION=us-east-1; DB=hl-m06-l3-pg

# 1. Protection settings and restore window
aws rds describe-db-instances --db-instance-identifier $DB \
  --query "DBInstances[0].{Retention:BackupRetentionPeriod,Window:PreferredBackupWindow,Latest:LatestRestorableTime,MultiAZ:MultiAZ}" \
  --profile $PROFILE --region $REGION

# 2. Raise retention to 7 days (non-zero -> non-zero: no outage)
aws rds modify-db-instance --db-instance-identifier $DB --backup-retention-period 7 --apply-immediately \
  --profile $PROFILE --region $REGION

# 3. Manual snapshot before a risky change (doesn't expire - delete it later)
aws rds create-db-snapshot --db-instance-identifier $DB \
  --db-snapshot-identifier hl-m06-l5-pre-release-$(date -u +%Y%m%d%H%M) \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m06-l5 --profile $PROFILE --region $REGION

# 4. Point-in-time restore to a NEW instance with the SAME network and parameter settings
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier $DB \
  --target-db-instance-identifier $DB-pitr \
  --restore-time 2026-11-18T16:39:00Z \
  --db-subnet-group-name hl-m06-l3 --vpc-security-group-ids sg-0ccccccccccccccc3 \
  --db-parameter-group-name hl-m06-l3-pg17 \
  --no-publicly-accessible --no-multi-az --enable-iam-database-authentication \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m06-l5 Key=Purpose,Value=restore-drill \
  --profile $PROFILE --region $REGION
aws rds wait db-instance-available --db-instance-identifier $DB-pitr --profile $PROFILE --region $REGION

# 5. Force a Multi-AZ failover (only if the instance is Multi-AZ)
aws rds reboot-db-instance --db-instance-identifier $DB --force-failover --profile $PROFILE --region $REGION

# 6. Read replica (async) and promotion - e.g. for a DR exercise
aws rds create-db-instance-read-replica --db-instance-identifier $DB-rr --source-db-instance-identifier $DB \
  --db-instance-class db.t4g.micro --no-publicly-accessible --profile $PROFILE --region $REGION
aws rds promote-read-replica --db-instance-identifier $DB-rr --profile $PROFILE --region $REGION

# 7. Cross-Region automated backups to us-west-2 (needs a KMS key in us-west-2 for encrypted sources)
aws rds start-db-instance-automated-backups-replication \
  --source-db-instance-arn arn:aws:rds:us-east-1:111122223333:db:$DB \
  --backup-retention-period 7 --kms-key-id arn:aws:kms:us-west-2:111122223333:key/EXAMPLE \
  --profile $PROFILE --region us-west-2

Note step 6: you can't create a read replica when the source uses RDS-managed master passwords in Secrets Manager (all engines except SQL Server and Db2). If your lab instance was created with --manage-master-user-password, skip step 6 or read the encryption and access lesson for the trade-off.

Infrastructure as Code

using Amazon.CDK;
using Amazon.CDK.AWS.Backup;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.RDS;
using Constructs;

public sealed class LedgerBackupStack : Stack
{
    public LedgerBackupStack(Construct scope, string id, IDatabaseInstance ledger, IStackProps? props = null) : base(scope, id, props)
    {
        // A dedicated vault with a lock: recovery points can't be deleted before MinRetention.
        var vault = new BackupVault(this, "Vault", new BackupVaultProps
        {
            BackupVaultName = "hl-m06-ledger-vault",
            LockConfiguration = new LockConfiguration { MinRetention = Duration.Days(35) },
            RemovalPolicy = RemovalPolicy.RETAIN,
        });

        var plan = new BackupPlan(this, "Plan", new BackupPlanProps { BackupVault = vault });
        plan.AddRule(new BackupPlanRule(new BackupPlanRuleProps
        {
            RuleName = "daily-35d",
            ScheduleExpression = Schedule.Cron(new CronOptions { Hour = "2", Minute = "0" }),
            DeleteAfter = Duration.Days(35),
            EnableContinuousBackup = true, // PITR via AWS Backup for supported engines ⚠ VERIFY
        }));
        plan.AddSelection("Ledger", new BackupSelectionOptions
        {
            Resources = new[] { BackupResource.FromRdsDatabaseInstance(ledger) },
        });
    }
}

Complete code

The restore drill orchestrator performs a point-in-time restore with the source's network and parameter settings, measures time to availability, writes an evidence file, and deletes the copy (unless --keep). Pair it with the ledger fingerprint command to verify the data.

RestoreDrill/RestoreDrill.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.RDS" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
  </ItemGroup>
</Project>
RestoreDrill/Program.cs
using System.Diagnostics;
using System.Text.Json;
using Amazon;
using Amazon.RDS;
using Amazon.RDS.Model;

// Restore drill orchestrator: point-in-time restore of an RDS instance into a NEW, private instance
// with the same network and parameter settings, measures how long it takes, writes an evidence file,
// and (unless --keep) deletes the restored copy.
//
// Usage: $env:AWS_PROFILE="learn-sandbox"; dotnet run -- <source-instance-id> [--keep] [--restore-time 2026-11-18T09:30:00Z]
// IAM: rds:DescribeDBInstances, rds:RestoreDBInstanceToPointInTime, rds:AddTagsToResource, rds:DeleteDBInstance

if (args.Length == 0)
{
    Console.Error.WriteLine("Usage: dotnet run -- <source-instance-id> [--keep] [--restore-time <UTC ISO-8601>]");
    return 2;
}

var sourceId = args[0];
var keep = args.Contains("--keep");
var restoreTimeArg = args.SkipWhile(a => a != "--restore-time").Skip(1).FirstOrDefault();
var targetId = $"{sourceId}-drill-{DateTime.UtcNow:HHmmss}";

using var rds = new AmazonRDSClient(RegionEndpoint.USEast1);

var source = (await rds.DescribeDBInstancesAsync(new DescribeDBInstancesRequest { DBInstanceIdentifier = sourceId })).DBInstances[0];
if ((source.BackupRetentionPeriod ?? 0) == 0)
{
    Console.Error.WriteLine($"{sourceId} has automated backups disabled: point-in-time restore is impossible.");
    return 1;
}

Console.WriteLine($"Source {sourceId}: earliest/latest restorable time = (see console) / {source.LatestRestorableTime:O}");

var request = new RestoreDBInstanceToPointInTimeRequest
{
    SourceDBInstanceIdentifier = sourceId,
    TargetDBInstanceIdentifier = targetId,
    DBInstanceClass = source.DBInstanceClass,
    DBSubnetGroupName = source.DBSubnetGroup.DBSubnetGroupName,
    VpcSecurityGroupIds = source.VpcSecurityGroups.Select(g => g.VpcSecurityGroupId).ToList(),
    // Without this, the restore gets the DEFAULT parameter group (e.g. no rds.force_ssl).
    DBParameterGroupName = source.DBParameterGroups[0].DBParameterGroupName,
    PubliclyAccessible = false,
    MultiAZ = false,
    EnableIAMDatabaseAuthentication = source.IAMDatabaseAuthenticationEnabled,
    DeletionProtection = false,
    Tags =
    [
        new Tag { Key = "Project", Value = "cea-lab" },
        new Tag { Key = "Lab", Value = "m06-l5" },
        new Tag { Key = "Environment", Value = "learn" },
        new Tag { Key = "Purpose", Value = "restore-drill" },
    ],
};

if (restoreTimeArg is null)
{
    request.UseLatestRestorableTime = true;
}
else
{
    request.RestoreTime = DateTime.Parse(restoreTimeArg, null, System.Globalization.DateTimeStyles.AdjustToUniversal);
}

var timer = Stopwatch.StartNew();
await rds.RestoreDBInstanceToPointInTimeAsync(request);
Console.WriteLine($"Restore of {targetId} started at {DateTime.UtcNow:O}");

DBInstance restored;
while (true)
{
    await Task.Delay(TimeSpan.FromSeconds(30));
    restored = (await rds.DescribeDBInstancesAsync(new DescribeDBInstancesRequest { DBInstanceIdentifier = targetId })).DBInstances[0];
    Console.WriteLine($"  {timer.Elapsed:mm\\:ss}  {restored.DBInstanceStatus}");
    if (restored.DBInstanceStatus == "available")
    {
        break;
    }

    if (restored.DBInstanceStatus is "failed" or "incompatible-restore" or "incompatible-parameters")
    {
        Console.Error.WriteLine($"Restore failed with status {restored.DBInstanceStatus}.");
        return 1;
    }
}

timer.Stop();
var evidence = new
{
    drill = "rds-point-in-time-restore",
    source = sourceId,
    target = targetId,
    requestedRestoreTime = restoreTimeArg ?? "latest",
    restoreMinutes = Math.Round(timer.Elapsed.TotalMinutes, 1),
    targetEndpoint = restored.Endpoint?.Address,
    encrypted = restored.StorageEncrypted,
    kmsKeyId = restored.KmsKeyId,
    parameterGroup = restored.DBParameterGroups[0].DBParameterGroupName,
    completedAtUtc = DateTime.UtcNow,
};
var file = $"restore-evidence-{DateTime.UtcNow:yyyyMMddTHHmmssZ}.json";
await File.WriteAllTextAsync(file, JsonSerializer.Serialize(evidence, new JsonSerializerOptions { WriteIndented = true }));
Console.WriteLine($"Available after {timer.Elapsed.TotalMinutes:F1} min. Evidence: {file}");
Console.WriteLine("Now verify the DATA: run the ledger `fingerprint` command against source and target and compare.");

if (!keep)
{
    await rds.DeleteDBInstanceAsync(new DeleteDBInstanceRequest
    {
        DBInstanceIdentifier = targetId,
        SkipFinalSnapshot = true,
        DeleteAutomatedBackups = true,
    });
    Console.WriteLine($"Deletion of {targetId} requested.");
}

return 0;

Verify the data with the module project (from a machine inside the VPC; --keep keeps the restored instance for this step):

dotnet run --project RestoreDrill -- hl-m06-ledger --keep --restore-time 2026-11-18T16:39:00Z

$env:Ledger__Database__Provider = "Postgres"
$env:Ledger__Database__AuthMode = "IamAuth"
$env:Ledger__Database__Username = "ledger_app"
$env:Ledger__Database__RootCertificatePath = "global-bundle.pem"
$env:Ledger__Database__Host = "<restored endpoint from the evidence file>"
dotnet run --project labs/m06-banking-data-store/src/Harbourline.Ledger.Api -- fingerprint

Code explained line by line

  1. The argument handling returns exit code 2 for usage errors, 1 for drill failures and 0 for success, so a scheduled pipeline can tell them apart.
  2. targetId = $"{sourceId}-drill-{...}": a unique, obviously temporary name. Drills must never reuse a production-looking identifier.
  3. DescribeDBInstancesAsync(...): reads the source's current settings. The restore copies them explicitly, because RDS doesn't copy parameter groups or security groups for you.
  4. BackupRetentionPeriod ?? 0) == 0: fails fast with a clear message. Discovering this during an incident is the worst time.
  5. DBSubnetGroupName, VpcSecurityGroupIds: keep the restored copy in the same private network, reachable by the same clients and only them.
  6. DBParameterGroupName: the lesson's opening story in one line. Without it, the restore gets the default group and loses settings such as rds.force_ssl.
  7. PubliclyAccessible = false, MultiAZ = false: a drill copy is private and single-AZ, which also keeps cost down.
  8. EnableIAMDatabaseAuthentication = source.IAMDatabaseAuthenticationEnabled: so the verification step can connect with the same IAM-authenticated user. Note the IAM policy for rds-db:connect names the DB resource ID, which differs on the restored instance, so the drill role needs a policy covering it.
  9. Tags: drill resources are tagged so the teardown check (and Cost Explorer) can find them.
  10. UseLatestRestorableTime vs RestoreTime: the two ways to choose the restore point. RestoreTime must fall inside the backup window and is in UTC.
  11. The polling loop: checks every 30 seconds and treats incompatible-* states as failures. In a production tool, add an overall timeout.
  12. evidence: duration, target time, encryption, key and parameter group, the facts an auditor asks for. Data verification is a separate, deliberate step (fingerprint).
  13. DeleteDBInstanceAsync(... SkipFinalSnapshot = true, DeleteAutomatedBackups = true): drill copies are disposable. Leaving them running is the most common drill cost overrun.

Security considerations

  • Restore = read access. Anyone who can restore a snapshot into an account they control can read the data. Restrict rds:Restore*, rds:CopyDBSnapshot and rds:ModifyDBSnapshotAttribute (sharing) to a small break-glass role. Alert on them.
  • Never share snapshots publicly. A snapshot can be made public with ModifyDBSnapshotAttribute. Detect it with IAM Access Analyzer (RDS snapshots are a supported resource type) and AWS Config rules.
  • Customer managed keys enable a backup account. To copy snapshots to a separate backup account, the snapshot must be encrypted with a CMK whose key policy allows that account. AWS managed keys can't be shared.
  • Immutable copies against ransomware: AWS Backup Vault Lock in compliance mode prevents anyone, including the account root user, from deleting recovery points before their retention ends. Use it for Tier 1 data.
  • Retention is a legal question too. Keeping backups longer helps recovery but conflicts with UK GDPR storage limitation and erasure requests. Agree retention with your DPO and document how erasure is handled in backups (typically: restored copies are re-scrubbed before use).
  • Drill copies are production data. Restore them only in the production account or an equally controlled DR account, never into a developer sandbox.

⚠ Learning shortcut — not for production

The lab deletes drill copies without a final snapshot and keeps 1–7 days of automated backups. A real Tier 1 setup uses 35-day retention, cross-Region automated backups, AWS Backup copies into a locked vault in a separate account, a documented restore runbook, and a monthly drill whose evidence is reviewed by someone outside the team.

Banking use case

Harbourline's monthly ledger restore drill (fictional):

  1. On the first Tuesday, a scheduled pipeline in the production account assumes the restore-drill role.
  2. It runs the orchestrator with --keep and a restore time of 06:00 that day, then runs fingerprint against the restore. It compares the result with the fingerprint the end-of-day job recorded at 06:00. The end-of-day job writes a fingerprint to an S3 evidence bucket with Object Lock every day.
  3. It also restores the latest AWS Backup recovery point from the backup account's locked vault into the DR account, proving the cross-account path works and that the KMS key policy is still correct.
  4. Results (restore minutes, fingerprint match, both paths) go to the operational resilience dashboard. Two consecutive failures trigger an incident.

During the "interest rate" incident, the team restores to 16:39 using the same orchestrator, verifies the fingerprint, and repairs the damaged rows from the restored copy using a reviewed SQL script. The production instance is never overwritten.

Common mistakes

  • Treating Multi-AZ or replicas as backups. They replicate damage.
  • Retention 0 "to save money". It disables PITR, and turning it back on causes an outage.
  • Restoring with defaults. Default parameter group, default security group and a public flag mean the restore is unreachable, insecure, or both.
  • Never testing restores. Untested backups are a hope, not a control.
  • Forgetting manual snapshots. They don't expire, count against a quota of 100, and cost money above the free allowance.
  • Deleting an instance and losing its automated backups. Choose Retain automated backups, or take a final snapshot.
  • Assuming cross-database consistency on SQL Server PITR. Databases are restored within one second of each other, so cross-database transactions may be inconsistent.
  • Read replicas with RDS-managed master secrets. Not supported for most engines. Plan the credential approach first.

Troubleshooting

SymptomLikely causeFix
LatestRestorableTime is hours oldInstance not available (for example storage-full), or SQL Server log chain brokenFix the instance state; for SQL Server check recovery model changes
Restore stuck in incompatible-restoreClass/storage/IOPS incompatible, or (SQL Server) too many databases for the classRestore with a compatible class and storage
App can't connect to the restored instanceDefault security group / parameter group usedSpecify SG and parameter group on restore
InvalidParameterCombination for a read replicaSource uses RDS-managed master passwordUse a different credential approach or skip replicas
SnapshotQuotaExceeded100 manual snapshots in the RegionDelete old snapshots; use AWS Backup (doesn't count)
Can't share a snapshot with the backup accountEncrypted with the AWS managed keyCopy with a CMK shared to the backup account

Cost and cleanup

  • Automated backups up to your provisioned storage size are free per Region. Manual snapshots and longer retention beyond that cost per GB-month.
  • A drill restore is a full instance for as long as it runs. For db.t4g.micro that is cents per hour ⚠ VERIFY, plus the time data loads in the background.
  • Cross-Region automated backups add storage in the second Region and data transfer charges.
  • A read replica is billed as a normal instance.

Teardown:

PROFILE=learn-sandbox; REGION=us-east-1; DB=hl-m06-l3-pg
aws rds delete-db-instance --db-instance-identifier $DB-pitr --skip-final-snapshot --delete-automated-backups --profile $PROFILE --region $REGION
aws rds delete-db-instance --db-instance-identifier $DB-rr --skip-final-snapshot --delete-automated-backups --profile $PROFILE --region $REGION 2>/dev/null
for s in $(aws rds describe-db-snapshots --snapshot-type manual --query "DBSnapshots[?starts_with(DBSnapshotIdentifier,'hl-m06-l5')].DBSnapshotIdentifier" --output text --profile $PROFILE --region $REGION); do
  aws rds delete-db-snapshot --db-snapshot-identifier "$s" --profile $PROFILE --region $REGION
done
aws rds stop-db-instance-automated-backups-replication --source-db-instance-arn arn:aws:rds:us-east-1:111122223333:db:$DB --profile $PROFILE --region us-west-2 2>/dev/null
# Then delete the source instance as in the RDS lesson, and verify nothing tagged Lab=m06-l5 remains:
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m06-l5 --query "ResourceTagMappingList[].ResourceARN" --profile $PROFILE --region $REGION

Free alternative: dotnet test --solution BankingDataStore.slnx in labs/m06-banking-data-store runs the local restore-and-fingerprint tests at no cost.

Exercises

  • 🟢 Guided: Take a manual snapshot, run the orchestrator with --keep against your lab instance, run fingerprint against source and restore, and compare. Delete everything.
  • 🟡 Challenge: Extend the orchestrator with a --from-snapshot <id> mode using RestoreDBInstanceFromDBSnapshotAsync, and add an overall timeout (for example 45 minutes) that fails the drill and still cleans up.
  • 🔴 Live problem: "Ransomware hit a partner bank. The board asks: if an attacker gets admin in our production AWS account, can they delete our ledger AND all its backups? Answer by Thursday, with what we'd change."

Assignment

Write docs/backup-and-restore-policy.md for the module project: tiers with RPO/RTO, the configuration per tier, retention (with the DPO caveat), the drill schedule, who reviews evidence, and a sample evidence JSON from your drill (fictional data).

Solutions

Knowledge check

Interview questions

  • "What's the difference between HA and backup?" HA (Multi-AZ, replicas) keeps the service running when infrastructure fails, and it copies every change, including mistakes. Backups and PITR give you a copy from before a mistake. A bank needs both, plus tested restores.
  • "How do you prove your backups work?" A scheduled restore drill: restore to a point in time with production settings, verify data with fingerprints or reconciliations against independently recorded values, measure time, keep evidence, and have someone independent review it.
  • "How do you protect backups from a compromised admin?" Cross-account copies with AWS Backup Vault Lock, CMKs shared with the backup account, SCPs denying deletions, deletion protection and alerts.
  • "Your RTO is 15 minutes but restores take an hour. What do you do?" Separate the scenarios. Infrastructure failure → HA (Multi-AZ/Aurora) meets 15 minutes. Data damage → negotiate a realistic RTO, or reduce it with faster restores (Aurora cloning or backtrack where applicable, smaller databases, pre-staged DR), and practise.

Certification practice

Completion checklist

  • I can say what each mechanism protects against, with its RPO and RTO
  • I know the retention range, defaults and which changes cause outages
  • I ran a restore drill (AWS or local) and verified data with a fingerprint
  • I deleted drill instances, replicas and manual snapshots, and stopped any backup replication
  • I wrote down an RPO/RTO tier table for the module project

Key takeaways

  • HA ≠ backup. Multi-AZ and replicas copy mistakes; only backups, snapshots and PITR undo them.
  • PITR restores to any second in a 0–35 day window, typically within ~5 minutes of now, always to a new instance with the settings you specify.
  • Restore with production settings: subnet group, security groups, parameter group, private, IAM auth.
  • Verify data, not just availability: fingerprints, reconciliations and evidence files.
  • Protect backups from your own admins: cross-account copies, Vault Lock, CMKs, SCPs.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page