Multi-AZ, read replicas, backups and point-in-time recovery
Know exactly what Multi-AZ instances, Multi-AZ DB clusters, read replicas, automated backups, snapshots and point-in-time restore protect against, map them to RPO and RTO, and run a repeatable, evidenced restore drill from C#.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.RDS 4.1.0.2) · AWS CLI v2 · RDS for PostgreSQL 17 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Say what each protection mechanism protects against: Multi-AZ DB instance, Multi-AZ DB cluster, read replicas, automated backups, manual snapshots, point-in-time restore (PITR), cross-Region automated backups and AWS Backup.
- Translate business requirements (RPO and RTO) into a configuration, and explain why "we have Multi-AZ" is not a backup strategy.
- Configure backup retention, windows and snapshots with the CLI and IaC, and know which changes cause an outage.
- Run a restore drill that measures time to restore and proves the restored data is complete, using a C# orchestrator and the module project's
fingerprintcommand. - Explain the RDS for SQL Server specifics: Always On Multi-AZ, Enterprise-only replicas, and per-database PITR consistency.
Prerequisites
- Amazon RDS for SQL Server and PostgreSQL
- Amazon Aurora, for comparison with Aurora's storage-level replication
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK. Optional: the module project inlabs/m06-banking-data-store
Why this matters in banking
At 16:40 on a Friday, a Harbourline release runs a data fix against the ledger. It contains an UPDATE with a missing WHERE clause, and every savings account's interest rate becomes 0.00%. The database is Multi-AZ, so the standby faithfully applies the same mistake within milliseconds. The read replica copies it a second later.
What saves the bank is point-in-time restore to 16:39, onto a new instance, followed by a careful repair of the damaged rows from the restored copy. Except that nobody had ever practised it: the team discovers mid-incident that the restored instance came up with the default parameter group and the wrong security group, so the application can't reach it. Two hours pass.
Under operational resilience rules, Harbourline must be able to show that important business services stay within their impact tolerance, and backups that have never been restored are not evidence. This lesson separates the mechanisms that protect against infrastructure failure from those that protect against data damage, and turns restores into a routine drill.
The idea in plain language
There are two different kinds of bad day:
- Something breaks: a disk, a host, an Availability Zone, a Region. You want the database back quickly with no data lost. Replication handles this: a standby or replica that already has the data.
- Something is damaged: a bad deployment, a malicious insider, ransomware, an accidental
DELETE. Replication makes this worse, because it copies the damage everywhere instantly. You need a copy from before the damage: backups, snapshots and point-in-time restore.
Two numbers describe what the business can tolerate:
- RPO (recovery point objective): how much data you may lose, measured in time. "RPO 5 minutes" means at most the last 5 minutes of changes.
- RTO (recovery time objective): how long the service may be down.
Analogy: Multi-AZ is a second teller standing next to the first, writing every entry into an identical ledger at the same moment. If the first teller faints, the second carries on. But if the first teller writes the wrong amount, so does the second. Backups are the photocopies of the ledger locked in the vault every night, plus the carbon copies of each day's slips. They are the only thing that can tell you what the ledger looked like before the mistake.
What problem each mechanism solves
| Mechanism | Protects against | Data loss (RPO) | Downtime (RTO) | Readable? | Notes |
|---|---|---|---|---|---|
| Multi-AZ DB instance (one standby) | Host, storage or AZ failure; also used during patching | ~0 (synchronous) | Typically 60–120 s failover | No, standby is not readable | DNS name stays the same; clients reconnect |
| Multi-AZ DB cluster (writer + 2 readable standbys, PostgreSQL/MySQL) | Same, plus read scaling | ~0 (semi-synchronous: one standby must acknowledge) | Usually faster than Multi-AZ instance ⚠ VERIFY current figures | Yes | Specific instance classes only (db.m6gd, db.r6gd, db.m8gd... with local NVMe) |
| Read replica (same or other Region) | Read load; can be promoted for DR | Replica lag (seconds+) | Minutes (manual promotion) | Yes | Asynchronous; up to 15 per primary; SQL Server: Enterprise only |
| Aurora replicas | Instance or AZ failure; read load | ~0 (shared storage) | Typically under 60 s, often under 30 s | Yes | See Aurora lesson |
| Automated backups + PITR | Data damage, deletion, corruption | Up to ~5 minutes (logs uploaded every 5 minutes) | Minutes to hours (new instance + data loading) | No | Retention 0–35 days; restore creates a new instance |
| Manual snapshots | Long-term restore points; before risky changes | Time of snapshot | Minutes to hours | No | Don't expire; 100 per Region quota; can be copied and shared |
| Cross-Region automated backups | Loss of the Region (for restores) | Few minutes | Hours (restore in another Region) | No | Snapshots and logs copied to a second Region |
| AWS Backup (with vault lock, cross-account copy) | Ransomware, insider deletion, policy compliance | Per plan | Hours | No | Central policies, immutable vaults, reporting |
Key facts behind the table (checked 2026-11-18):
- Retention can be 0–35 days for DB instances. 0 disables automated backups and PITR. The console default is 7 days; the API/CLI default is 1 day. Multi-AZ DB clusters need 1–35.
- Changing retention between 0 and non-zero causes an outage. Changing between two non-zero values doesn't.
- RDS uploads transaction logs to S3 every five minutes, so
LatestRestorableTimeis typically within about five minutes of now. - Restores always create a new instance. A restored instance gets the default parameter and option groups unless you specify others on the API/CLI. Console restores let you change them afterwards.
- Free backup storage equals 100% of your provisioned database storage per Region. Beyond that you pay per GB-month.
- Snapshots of an encrypted instance stay encrypted with the same key. You can choose a different key when copying a snapshot.
- If you delete an instance without Retain automated backups, its automated backups go with it. Manual snapshots survive instance deletion.
- SQL Server: PITR restores each database in the instance to within one second of each other, so transactions spanning databases may be inconsistent. Changing a database's recovery model (for example to
BULK_LOGGED) can break the log chain. SQL Server read replicas don't support PITR on the replica itself.
Why Multi-AZ is not a backup
Multi-AZ replicates every change, including your mistakes, within the same commit. It protects availability, not integrity. Every bank database therefore needs both high availability and a tested restore path.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Always On AG synchronous secondary (no read) | Multi-AZ DB instance | Managed; standby unreadable; you don't see the cluster |
| AG readable secondaries | Multi-AZ DB cluster (PG/MySQL) or read replicas | Readable standbys need specific instance classes |
| Full + differential + log backups, maintenance plans | Automated backups (daily snapshot + 5-minute log upload) | No backup jobs to write; retention is a setting |
RESTORE ... WITH STOPAT | Point-in-time restore to a new instance | You can't restore "over" the existing instance |
BACKUP DATABASE ... COPY_ONLY before a release | Manual snapshot | Instance-level, not per database |
| Azure SQL LTR (long-term retention) | Manual snapshots / AWS Backup plans / snapshot export to S3 | AWS Backup adds vault lock and cross-account copy |
| Geo-restore from geo-redundant backups | Cross-Region automated backups | You enable and pay for it per instance |
| Azure Backup immutable vaults | AWS Backup Vault Lock (compliance mode) | Same idea: nobody, including root, can delete early |
Architecture
Security-boundary view.
- Snapshots and automated backups are encrypted with the instance's KMS key. Copies to another account need a customer managed key shared with that account.
- The backup account is a separate AWS account with its own IAM and a locked vault. A compromised production account can't delete those copies.
- Restore permissions (
rds:RestoreDBInstanceToPointInTime,rds:RestoreDBInstanceFromDBSnapshot) are powerful: whoever can restore can read the data. Grant them only to a break-glass or DBA role, and alert on their use with CloudTrail.
Mapping requirements to configuration
| Service tier (fictional Harbourline policy) | RPO | RTO | Configuration |
|---|---|---|---|
| Tier 1: ledger, payments | ≤ 5 min (data damage); ~0 (infrastructure) | ≤ 15 min (infrastructure); ≤ 4 h (data damage) | Aurora or Multi-AZ; 35-day retention; cross-Region automated backups; AWS Backup copies to a locked vault in the backup account; monthly restore drill |
| Tier 2: customer servicing | ≤ 15 min | ≤ 4 h | Multi-AZ instance; 14-day retention; AWS Backup daily; quarterly drill |
| Tier 3: internal tools | ≤ 24 h | ≤ 1 business day | Single-AZ; 7-day retention; annual drill |
Step-by-step implementation
The drill below uses the lab instance from the RDS lesson, or the module project's hl-m06-ledger instance. If you prefer the free path, follow the local version at the end of this section.
Check the protection settings
Confirm the instance has backup retention of at least 1 (7+ in production), and note LatestRestorableTime. If retention is 0, there is nothing to restore from.
Take a manual snapshot before a "risky change"
Banks take a named snapshot before every schema migration or data fix. The snapshot doesn't expire, so delete it when the change is signed off.
Simulate damage
In a test schema, record the time, then run a deliberately bad UPDATE (on fictional data only). Note the exact UTC time just before the damage. This is your restore target.
Restore to a point in time, correctly
Restore to a new instance with the same subnet group, security groups and parameter group, not publicly accessible, with IAM authentication on, and with drill tags. The C# orchestrator in Complete code does exactly this and measures the time.
Verify the data, not just the instance
An "available" instance proves only that RDS worked. Run the ledger fingerprint command (in labs/m06-banking-data-store) against the restored instance. It prints counts, totals and a SHA-256 digest over every transaction. Compare it with the expected state, for example a fingerprint taken just before the damage. The module project's scripts/restore-drill.sh automates source-vs-restore comparison.
Record evidence and clean up
Keep the evidence file (restore duration, target time, fingerprint comparison). Delete the restored instance and the manual snapshot. File the evidence where your operational resilience team expects it.
Free local version: in labs/m06-banking-data-store, dotnet test includes RestoreFingerprintTests. One test backs up a seeded SQLite ledger with SQLite's online backup API, "restores" it into a fresh database and proves the fingerprints match. Another simulates a restore point taken before a posting and proves the fingerprint detects the missing transaction. The mechanics of measuring, verifying and recording are identical.
AWS Console instructions
- Console → RDS → Databases → your instance → Maintenance & backups: see the retention period, backup window and latest restore time. To change retention: Modify → Backup → Backup retention period.
- Manual snapshot: select the instance, then Actions → Take snapshot. Name it
hl-m06-l5-pre-release-<date>. - PITR: Console → RDS → Automated backups → select the instance → Actions → Restore to point in time. Choose Custom and enter the time (shown in your local time zone; check the UTC offset). Then set the identifier, DB subnet group, Public access: No, the security group and the parameter group (under Additional configuration).
- Cross-Region automated backups: Automated backups → Replicated tab → Manage cross-Region replication (or from the instance's Actions menu ⚠ VERIFY the current menu location).
- Failover test (Multi-AZ): Actions → Reboot → Reboot with failover.
AWS CLI
PROFILE=learn-sandbox; REGION=us-east-1; DB=hl-m06-l3-pg
# 1. Protection settings and restore window
aws rds describe-db-instances --db-instance-identifier $DB \
--query "DBInstances[0].{Retention:BackupRetentionPeriod,Window:PreferredBackupWindow,Latest:LatestRestorableTime,MultiAZ:MultiAZ}" \
--profile $PROFILE --region $REGION
# 2. Raise retention to 7 days (non-zero -> non-zero: no outage)
aws rds modify-db-instance --db-instance-identifier $DB --backup-retention-period 7 --apply-immediately \
--profile $PROFILE --region $REGION
# 3. Manual snapshot before a risky change (doesn't expire - delete it later)
aws rds create-db-snapshot --db-instance-identifier $DB \
--db-snapshot-identifier hl-m06-l5-pre-release-$(date -u +%Y%m%d%H%M) \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m06-l5 --profile $PROFILE --region $REGION
# 4. Point-in-time restore to a NEW instance with the SAME network and parameter settings
aws rds restore-db-instance-to-point-in-time \
--source-db-instance-identifier $DB \
--target-db-instance-identifier $DB-pitr \
--restore-time 2026-11-18T16:39:00Z \
--db-subnet-group-name hl-m06-l3 --vpc-security-group-ids sg-0ccccccccccccccc3 \
--db-parameter-group-name hl-m06-l3-pg17 \
--no-publicly-accessible --no-multi-az --enable-iam-database-authentication \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m06-l5 Key=Purpose,Value=restore-drill \
--profile $PROFILE --region $REGION
aws rds wait db-instance-available --db-instance-identifier $DB-pitr --profile $PROFILE --region $REGION
# 5. Force a Multi-AZ failover (only if the instance is Multi-AZ)
aws rds reboot-db-instance --db-instance-identifier $DB --force-failover --profile $PROFILE --region $REGION
# 6. Read replica (async) and promotion - e.g. for a DR exercise
aws rds create-db-instance-read-replica --db-instance-identifier $DB-rr --source-db-instance-identifier $DB \
--db-instance-class db.t4g.micro --no-publicly-accessible --profile $PROFILE --region $REGION
aws rds promote-read-replica --db-instance-identifier $DB-rr --profile $PROFILE --region $REGION
# 7. Cross-Region automated backups to us-west-2 (needs a KMS key in us-west-2 for encrypted sources)
aws rds start-db-instance-automated-backups-replication \
--source-db-instance-arn arn:aws:rds:us-east-1:111122223333:db:$DB \
--backup-retention-period 7 --kms-key-id arn:aws:kms:us-west-2:111122223333:key/EXAMPLE \
--profile $PROFILE --region us-west-2Note step 6: you can't create a read replica when the source uses RDS-managed master passwords in Secrets Manager (all engines except SQL Server and Db2). If your lab instance was created with --manage-master-user-password, skip step 6 or read the encryption and access lesson for the trade-off.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.Backup;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.RDS;
using Constructs;
public sealed class LedgerBackupStack : Stack
{
public LedgerBackupStack(Construct scope, string id, IDatabaseInstance ledger, IStackProps? props = null) : base(scope, id, props)
{
// A dedicated vault with a lock: recovery points can't be deleted before MinRetention.
var vault = new BackupVault(this, "Vault", new BackupVaultProps
{
BackupVaultName = "hl-m06-ledger-vault",
LockConfiguration = new LockConfiguration { MinRetention = Duration.Days(35) },
RemovalPolicy = RemovalPolicy.RETAIN,
});
var plan = new BackupPlan(this, "Plan", new BackupPlanProps { BackupVault = vault });
plan.AddRule(new BackupPlanRule(new BackupPlanRuleProps
{
RuleName = "daily-35d",
ScheduleExpression = Schedule.Cron(new CronOptions { Hour = "2", Minute = "0" }),
DeleteAfter = Duration.Days(35),
EnableContinuousBackup = true, // PITR via AWS Backup for supported engines ⚠ VERIFY
}));
plan.AddSelection("Ledger", new BackupSelectionOptions
{
Resources = new[] { BackupResource.FromRdsDatabaseInstance(ledger) },
});
}
}Complete code
The restore drill orchestrator performs a point-in-time restore with the source's network and parameter settings, measures time to availability, writes an evidence file, and deletes the copy (unless --keep). Pair it with the ledger fingerprint command to verify the data.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.RDS" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
</ItemGroup>
</Project>using System.Diagnostics;
using System.Text.Json;
using Amazon;
using Amazon.RDS;
using Amazon.RDS.Model;
// Restore drill orchestrator: point-in-time restore of an RDS instance into a NEW, private instance
// with the same network and parameter settings, measures how long it takes, writes an evidence file,
// and (unless --keep) deletes the restored copy.
//
// Usage: $env:AWS_PROFILE="learn-sandbox"; dotnet run -- <source-instance-id> [--keep] [--restore-time 2026-11-18T09:30:00Z]
// IAM: rds:DescribeDBInstances, rds:RestoreDBInstanceToPointInTime, rds:AddTagsToResource, rds:DeleteDBInstance
if (args.Length == 0)
{
Console.Error.WriteLine("Usage: dotnet run -- <source-instance-id> [--keep] [--restore-time <UTC ISO-8601>]");
return 2;
}
var sourceId = args[0];
var keep = args.Contains("--keep");
var restoreTimeArg = args.SkipWhile(a => a != "--restore-time").Skip(1).FirstOrDefault();
var targetId = $"{sourceId}-drill-{DateTime.UtcNow:HHmmss}";
using var rds = new AmazonRDSClient(RegionEndpoint.USEast1);
var source = (await rds.DescribeDBInstancesAsync(new DescribeDBInstancesRequest { DBInstanceIdentifier = sourceId })).DBInstances[0];
if ((source.BackupRetentionPeriod ?? 0) == 0)
{
Console.Error.WriteLine($"{sourceId} has automated backups disabled: point-in-time restore is impossible.");
return 1;
}
Console.WriteLine($"Source {sourceId}: earliest/latest restorable time = (see console) / {source.LatestRestorableTime:O}");
var request = new RestoreDBInstanceToPointInTimeRequest
{
SourceDBInstanceIdentifier = sourceId,
TargetDBInstanceIdentifier = targetId,
DBInstanceClass = source.DBInstanceClass,
DBSubnetGroupName = source.DBSubnetGroup.DBSubnetGroupName,
VpcSecurityGroupIds = source.VpcSecurityGroups.Select(g => g.VpcSecurityGroupId).ToList(),
// Without this, the restore gets the DEFAULT parameter group (e.g. no rds.force_ssl).
DBParameterGroupName = source.DBParameterGroups[0].DBParameterGroupName,
PubliclyAccessible = false,
MultiAZ = false,
EnableIAMDatabaseAuthentication = source.IAMDatabaseAuthenticationEnabled,
DeletionProtection = false,
Tags =
[
new Tag { Key = "Project", Value = "cea-lab" },
new Tag { Key = "Lab", Value = "m06-l5" },
new Tag { Key = "Environment", Value = "learn" },
new Tag { Key = "Purpose", Value = "restore-drill" },
],
};
if (restoreTimeArg is null)
{
request.UseLatestRestorableTime = true;
}
else
{
request.RestoreTime = DateTime.Parse(restoreTimeArg, null, System.Globalization.DateTimeStyles.AdjustToUniversal);
}
var timer = Stopwatch.StartNew();
await rds.RestoreDBInstanceToPointInTimeAsync(request);
Console.WriteLine($"Restore of {targetId} started at {DateTime.UtcNow:O}");
DBInstance restored;
while (true)
{
await Task.Delay(TimeSpan.FromSeconds(30));
restored = (await rds.DescribeDBInstancesAsync(new DescribeDBInstancesRequest { DBInstanceIdentifier = targetId })).DBInstances[0];
Console.WriteLine($" {timer.Elapsed:mm\\:ss} {restored.DBInstanceStatus}");
if (restored.DBInstanceStatus == "available")
{
break;
}
if (restored.DBInstanceStatus is "failed" or "incompatible-restore" or "incompatible-parameters")
{
Console.Error.WriteLine($"Restore failed with status {restored.DBInstanceStatus}.");
return 1;
}
}
timer.Stop();
var evidence = new
{
drill = "rds-point-in-time-restore",
source = sourceId,
target = targetId,
requestedRestoreTime = restoreTimeArg ?? "latest",
restoreMinutes = Math.Round(timer.Elapsed.TotalMinutes, 1),
targetEndpoint = restored.Endpoint?.Address,
encrypted = restored.StorageEncrypted,
kmsKeyId = restored.KmsKeyId,
parameterGroup = restored.DBParameterGroups[0].DBParameterGroupName,
completedAtUtc = DateTime.UtcNow,
};
var file = $"restore-evidence-{DateTime.UtcNow:yyyyMMddTHHmmssZ}.json";
await File.WriteAllTextAsync(file, JsonSerializer.Serialize(evidence, new JsonSerializerOptions { WriteIndented = true }));
Console.WriteLine($"Available after {timer.Elapsed.TotalMinutes:F1} min. Evidence: {file}");
Console.WriteLine("Now verify the DATA: run the ledger `fingerprint` command against source and target and compare.");
if (!keep)
{
await rds.DeleteDBInstanceAsync(new DeleteDBInstanceRequest
{
DBInstanceIdentifier = targetId,
SkipFinalSnapshot = true,
DeleteAutomatedBackups = true,
});
Console.WriteLine($"Deletion of {targetId} requested.");
}
return 0;Verify the data with the module project (from a machine inside the VPC; --keep keeps the restored instance for this step):
dotnet run --project RestoreDrill -- hl-m06-ledger --keep --restore-time 2026-11-18T16:39:00Z
$env:Ledger__Database__Provider = "Postgres"
$env:Ledger__Database__AuthMode = "IamAuth"
$env:Ledger__Database__Username = "ledger_app"
$env:Ledger__Database__RootCertificatePath = "global-bundle.pem"
$env:Ledger__Database__Host = "<restored endpoint from the evidence file>"
dotnet run --project labs/m06-banking-data-store/src/Harbourline.Ledger.Api -- fingerprintCode explained line by line
- The argument handling returns exit code 2 for usage errors, 1 for drill failures and 0 for success, so a scheduled pipeline can tell them apart.
targetId = $"{sourceId}-drill-{...}": a unique, obviously temporary name. Drills must never reuse a production-looking identifier.DescribeDBInstancesAsync(...): reads the source's current settings. The restore copies them explicitly, because RDS doesn't copy parameter groups or security groups for you.BackupRetentionPeriod ?? 0) == 0: fails fast with a clear message. Discovering this during an incident is the worst time.DBSubnetGroupName,VpcSecurityGroupIds: keep the restored copy in the same private network, reachable by the same clients and only them.DBParameterGroupName: the lesson's opening story in one line. Without it, the restore gets the default group and loses settings such asrds.force_ssl.PubliclyAccessible = false,MultiAZ = false: a drill copy is private and single-AZ, which also keeps cost down.EnableIAMDatabaseAuthentication = source.IAMDatabaseAuthenticationEnabled: so the verification step can connect with the same IAM-authenticated user. Note the IAM policy forrds-db:connectnames the DB resource ID, which differs on the restored instance, so the drill role needs a policy covering it.Tags: drill resources are tagged so the teardown check (and Cost Explorer) can find them.UseLatestRestorableTimevsRestoreTime: the two ways to choose the restore point.RestoreTimemust fall inside the backup window and is in UTC.- The polling loop: checks every 30 seconds and treats
incompatible-*states as failures. In a production tool, add an overall timeout. evidence: duration, target time, encryption, key and parameter group, the facts an auditor asks for. Data verification is a separate, deliberate step (fingerprint).DeleteDBInstanceAsync(... SkipFinalSnapshot = true, DeleteAutomatedBackups = true): drill copies are disposable. Leaving them running is the most common drill cost overrun.
Security considerations
- Restore = read access. Anyone who can restore a snapshot into an account they control can read the data. Restrict
rds:Restore*,rds:CopyDBSnapshotandrds:ModifyDBSnapshotAttribute(sharing) to a small break-glass role. Alert on them. - Never share snapshots publicly. A snapshot can be made public with
ModifyDBSnapshotAttribute. Detect it with IAM Access Analyzer (RDS snapshots are a supported resource type) and AWS Config rules. - Customer managed keys enable a backup account. To copy snapshots to a separate backup account, the snapshot must be encrypted with a CMK whose key policy allows that account. AWS managed keys can't be shared.
- Immutable copies against ransomware: AWS Backup Vault Lock in compliance mode prevents anyone, including the account root user, from deleting recovery points before their retention ends. Use it for Tier 1 data.
- Retention is a legal question too. Keeping backups longer helps recovery but conflicts with UK GDPR storage limitation and erasure requests. Agree retention with your DPO and document how erasure is handled in backups (typically: restored copies are re-scrubbed before use).
- Drill copies are production data. Restore them only in the production account or an equally controlled DR account, never into a developer sandbox.
⚠ Learning shortcut — not for production
The lab deletes drill copies without a final snapshot and keeps 1–7 days of automated backups. A real Tier 1 setup uses 35-day retention, cross-Region automated backups, AWS Backup copies into a locked vault in a separate account, a documented restore runbook, and a monthly drill whose evidence is reviewed by someone outside the team.
Banking use case
Harbourline's monthly ledger restore drill (fictional):
- On the first Tuesday, a scheduled pipeline in the production account assumes the
restore-drillrole. - It runs the orchestrator with
--keepand a restore time of 06:00 that day, then runsfingerprintagainst the restore. It compares the result with the fingerprint the end-of-day job recorded at 06:00. The end-of-day job writes a fingerprint to an S3 evidence bucket with Object Lock every day. - It also restores the latest AWS Backup recovery point from the backup account's locked vault into the DR account, proving the cross-account path works and that the KMS key policy is still correct.
- Results (restore minutes, fingerprint match, both paths) go to the operational resilience dashboard. Two consecutive failures trigger an incident.
During the "interest rate" incident, the team restores to 16:39 using the same orchestrator, verifies the fingerprint, and repairs the damaged rows from the restored copy using a reviewed SQL script. The production instance is never overwritten.
Common mistakes
- Treating Multi-AZ or replicas as backups. They replicate damage.
- Retention 0 "to save money". It disables PITR, and turning it back on causes an outage.
- Restoring with defaults. Default parameter group, default security group and a public flag mean the restore is unreachable, insecure, or both.
- Never testing restores. Untested backups are a hope, not a control.
- Forgetting manual snapshots. They don't expire, count against a quota of 100, and cost money above the free allowance.
- Deleting an instance and losing its automated backups. Choose Retain automated backups, or take a final snapshot.
- Assuming cross-database consistency on SQL Server PITR. Databases are restored within one second of each other, so cross-database transactions may be inconsistent.
- Read replicas with RDS-managed master secrets. Not supported for most engines. Plan the credential approach first.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
LatestRestorableTime is hours old | Instance not available (for example storage-full), or SQL Server log chain broken | Fix the instance state; for SQL Server check recovery model changes |
Restore stuck in incompatible-restore | Class/storage/IOPS incompatible, or (SQL Server) too many databases for the class | Restore with a compatible class and storage |
| App can't connect to the restored instance | Default security group / parameter group used | Specify SG and parameter group on restore |
InvalidParameterCombination for a read replica | Source uses RDS-managed master password | Use a different credential approach or skip replicas |
SnapshotQuotaExceeded | 100 manual snapshots in the Region | Delete old snapshots; use AWS Backup (doesn't count) |
| Can't share a snapshot with the backup account | Encrypted with the AWS managed key | Copy with a CMK shared to the backup account |
Cost and cleanup
- Automated backups up to your provisioned storage size are free per Region. Manual snapshots and longer retention beyond that cost per GB-month.
- A drill restore is a full instance for as long as it runs. For
db.t4g.microthat is cents per hour ⚠ VERIFY, plus the time data loads in the background. - Cross-Region automated backups add storage in the second Region and data transfer charges.
- A read replica is billed as a normal instance.
Teardown:
PROFILE=learn-sandbox; REGION=us-east-1; DB=hl-m06-l3-pg
aws rds delete-db-instance --db-instance-identifier $DB-pitr --skip-final-snapshot --delete-automated-backups --profile $PROFILE --region $REGION
aws rds delete-db-instance --db-instance-identifier $DB-rr --skip-final-snapshot --delete-automated-backups --profile $PROFILE --region $REGION 2>/dev/null
for s in $(aws rds describe-db-snapshots --snapshot-type manual --query "DBSnapshots[?starts_with(DBSnapshotIdentifier,'hl-m06-l5')].DBSnapshotIdentifier" --output text --profile $PROFILE --region $REGION); do
aws rds delete-db-snapshot --db-snapshot-identifier "$s" --profile $PROFILE --region $REGION
done
aws rds stop-db-instance-automated-backups-replication --source-db-instance-arn arn:aws:rds:us-east-1:111122223333:db:$DB --profile $PROFILE --region us-west-2 2>/dev/null
# Then delete the source instance as in the RDS lesson, and verify nothing tagged Lab=m06-l5 remains:
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m06-l5 --query "ResourceTagMappingList[].ResourceARN" --profile $PROFILE --region $REGIONFree alternative: dotnet test --solution BankingDataStore.slnx in labs/m06-banking-data-store runs the local restore-and-fingerprint tests at no cost.
Exercises
- 🟢 Guided: Take a manual snapshot, run the orchestrator with
--keepagainst your lab instance, runfingerprintagainst source and restore, and compare. Delete everything. - 🟡 Challenge: Extend the orchestrator with a
--from-snapshot <id>mode usingRestoreDBInstanceFromDBSnapshotAsync, and add an overall timeout (for example 45 minutes) that fails the drill and still cleans up. - 🔴 Live problem: "Ransomware hit a partner bank. The board asks: if an attacker gets admin in our production AWS account, can they delete our ledger AND all its backups? Answer by Thursday, with what we'd change."
Assignment
Write docs/backup-and-restore-policy.md for the module project: tiers with RPO/RTO, the configuration per tier, retention (with the DPO caveat), the drill schedule, who reviews evidence, and a sample evidence JSON from your drill (fictional data).
Solutions
On a quiet lab instance, the source and restore fingerprints match exactly. The restore point is within about five minutes of now, and nothing was written since. If you posted transactions after the restore point, the restore lacks them: Transactions, BalanceTotal and TransactionDigest differ, while LatestTransactionId on the restore equals the last transaction before the restore time.
Why fingerprints: an "available" instance proves RDS restored something. Counts, totals and a digest over every transaction prove the ledger came back as expected.
Add a branch:
if (snapshotId is not null)
{
await rds.RestoreDBInstanceFromDBSnapshotAsync(new RestoreDBInstanceFromDBSnapshotRequest
{
DBSnapshotIdentifier = snapshotId,
DBInstanceIdentifier = targetId,
DBInstanceClass = source.DBInstanceClass,
DBSubnetGroupName = source.DBSubnetGroup.DBSubnetGroupName,
VpcSecurityGroupIds = source.VpcSecurityGroups.Select(g => g.VpcSecurityGroupId).ToList(),
DBParameterGroupName = source.DBParameterGroups[0].DBParameterGroupName,
PubliclyAccessible = false,
Tags = request.Tags,
});
}For the timeout, create using var deadline = new CancellationTokenSource(TimeSpan.FromMinutes(45));, pass it to Task.Delay, and wrap the loop in try/finally so the delete runs even on timeout or failure. Why: a drill that leaves a stray instance behind costs money and leaves a copy of production data lying around.
Honest answer: today, probably yes. An admin in the production account can delete the instance, its automated backups (by choosing not to retain them), manual snapshots and in-account AWS Backup recovery points, unless they are protected.
Changes:
- AWS Backup copies to a separate backup account with Vault Lock in compliance mode. Nobody in the production account, or even the backup account's root user, can delete them before retention ends.
- A CMK for the ledger whose key policy allows the backup account. Also hold a copy of key access in the backup account, so the production admin can't brick restores by disabling the key.
- SCPs (Module 5) that deny
rds:DeleteDBInstancewithout a final snapshot, denybackup:DeleteRecoveryPoint, and deny disabling deletion protection, except for a break-glass role. - Deletion protection on and cross-Region automated backups.
- Alerts on deletion and on attempts to change key policies.
- Restore the locked copies in the monthly drill, to prove they're usable.
Knowledge check
Answer: Multi-AZ replicates every committed change synchronously to the standby, including the DELETE. It protects against infrastructure failure, not data damage. You need PITR, snapshots or AWS Backup for that.
Answer: 0–35 days for DB instances (1–35 for Multi-AZ DB clusters). 0 disables automated backups and PITR. The console defaults to 7 days, the CLI/API to 1 day.
Answer: The restored instance uses the default DB parameter group (and default option group). Specify your custom groups during restore to keep settings such as forced TLS.
Answer: Only the Multi-AZ DB cluster standbys (two readers, PostgreSQL/MySQL, specific instance classes). A Multi-AZ DB instance standby can't serve reads.
Answer: About five minutes, because RDS uploads transaction logs to S3 every five minutes.
Interview questions
- "What's the difference between HA and backup?" HA (Multi-AZ, replicas) keeps the service running when infrastructure fails, and it copies every change, including mistakes. Backups and PITR give you a copy from before a mistake. A bank needs both, plus tested restores.
- "How do you prove your backups work?" A scheduled restore drill: restore to a point in time with production settings, verify data with fingerprints or reconciliations against independently recorded values, measure time, keep evidence, and have someone independent review it.
- "How do you protect backups from a compromised admin?" Cross-account copies with AWS Backup Vault Lock, CMKs shared with the backup account, SCPs denying deletions, deletion protection and alerts.
- "Your RTO is 15 minutes but restores take an hour. What do you do?" Separate the scenarios. Infrastructure failure → HA (Multi-AZ/Aurora) meets 15 minutes. Data damage → negotiate a realistic RTO, or reduce it with faster restores (Aurora cloning or backtrack where applicable, smaller databases, pre-staged DR), and practise.
Certification practice
A developer accidentally ran an UPDATE without a WHERE clause on an Amazon RDS for PostgreSQL Multi-AZ DB instance 20 minutes ago. Automated backups are enabled with 7-day retention. The company must recover the data as it was just before the update. What should a solutions architect do?
A. Fail over to the standby instance B. Promote a read replica C. Restore the DB instance to a point in time just before the update, as a new DB instance, and copy the correct data back D. Restore the latest automated snapshot over the existing instance
Answer: C. PITR restores to any second in the retention window, creating a new instance. A and B contain the same bad data, because replication copied it. D is wrong twice: restores never overwrite an existing instance, and the latest daily snapshot may predate or postdate the change without second-level precision.
A company needs automatic failover across Availability Zones for RDS for PostgreSQL, and wants the standby capacity to serve read queries as well. Which deployment meets these requirements?
A. Multi-AZ DB instance deployment B. Multi-AZ DB cluster deployment C. Single-AZ instance with a read replica in another AZ D. Single-AZ instance with cross-Region automated backups
Answer: B. A Multi-AZ DB cluster has a writer and two readable standbys in three AZs with automatic failover. A's standby isn't readable. C's replica is asynchronous, and promotion isn't automatic. D is for restores, not failover.
A financial institution must ensure that database backups cannot be deleted by anyone, including administrators, for 35 days, to protect against ransomware. Which solution meets this requirement?
A. Set the RDS backup retention period to 35 days B. Use AWS Backup with a backup vault protected by AWS Backup Vault Lock in compliance mode C. Enable deletion protection on the DB instance D. Copy manual snapshots to another Region every day
Answer: B. Vault Lock in compliance mode prevents deletion of recovery points by anyone before their retention period ends. A can be shortened by an admin, and the backups can be deleted with the instance. C protects the instance, not its backups. D's copies can still be deleted by an admin.
Completion checklist
- I can say what each mechanism protects against, with its RPO and RTO
- I know the retention range, defaults and which changes cause outages
- I ran a restore drill (AWS or local) and verified data with a fingerprint
- I deleted drill instances, replicas and manual snapshots, and stopped any backup replication
- I wrote down an RPO/RTO tier table for the module project
Key takeaways
- HA ≠ backup. Multi-AZ and replicas copy mistakes; only backups, snapshots and PITR undo them.
- PITR restores to any second in a 0–35 day window, typically within ~5 minutes of now, always to a new instance with the settings you specify.
- Restore with production settings: subnet group, security groups, parameter group, private, IAM auth.
- Verify data, not just availability: fingerprints, reconciliations and evidence files.
- Protect backups from your own admins: cross-account copies, Vault Lock, CMKs, SCPs.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Amazon Aurora: storage, endpoints, Serverless v2 and Global Database
Understand how Aurora separates compute from a six-copy, three-AZ storage volume, use writer and reader endpoints from .NET, size Aurora Serverless v2, measure a real failover, and design cross-Region DR with Aurora Global Database.
Encryption and database access
Encrypt an RDS database at rest with a customer managed KMS key and in transit with verified TLS, then choose between IAM database authentication and rotated Secrets Manager credentials, and prove from C# what the database says about your session.