AWS Backup in depth
Build a backup plan with vaults, Vault Lock, cross-Region and cross-account copies, and automated restore testing with validation, then produce restore evidence a bank can show an auditor.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Backup, AWSSDK.DynamoDBv2) · AWS CLI v2 · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Region us-east-1 (copy target us-west-2). Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Build an AWS Backup plan with schedule, lifecycle, tag-based resource selection and a cross-Region copy.
- Explain vault access policies, Vault Lock (governance and compliance mode) and why compliance mode is irreversible after its grace time.
- Design a cross-account backup so that a compromised production account cannot delete its own backups.
- Configure restore testing with a validation step, and write a C# Lambda that validates a restored DynamoDB table and reports the result.
- Turn restore job timings into evidence against an RTO.
Prerequisites
- Multi-AZ, read replicas, backups and point-in-time recovery
- Backup security and key rotation for the KMS and ransomware angle
- Database failover across Regions (replication versus backup)
- Tools: AWS CLI v2 (
learn-sandbox), .NET 10 SDK
Why this matters in banking
A bank's worst day is not always a dead data centre. It can be a ransomware actor who has gained administrator rights in the production account, found the backups, and deleted them before encrypting the data. Replication does not help (it copies the damage) and neither do backups that the same administrator can delete.
The control that matters is immutable, separated backups: copies in a different account, in a vault that even the account's administrators cannot empty before the retention period ends, plus proof that you can restore from them. Operational-resilience regimes ask for exactly that kind of evidence: not "we have backups", but "we restored last month's backup in 22 minutes and the data checked out".
The idea in plain language
AWS Backup is a central service that takes scheduled backups of many different AWS resources using one policy language. Instead of each team configuring RDS snapshots, DynamoDB backups, EBS snapshots and EFS backups differently, you write a plan: when to back up, how long to keep, where to copy. Resources are attached to the plan by tags or ARNs.
The vocabulary:
- Backup vault: a container for backups (recovery points), encrypted with a KMS key, with an access policy.
- Recovery point: one backup of one resource.
- Backup plan: rules (schedule, window, lifecycle, copy actions).
- Selection (resource assignment): which resources the plan covers, usually by tag.
- Vault Lock: a write-once-read-many (WORM) setting that stops anyone deleting recovery points before their retention ends.
- Restore testing: scheduled automatic restores of recent recovery points, with optional validation, measuring how long each takes.
Analogy: The backup plan is the bank's cash-in-transit schedule, a vault is the secure room it goes to, and Vault Lock is a time lock that nobody, including the branch manager, can open early. Restore testing is the monthly audit where you really open a box and count the notes.
What problem AWS Backup solves
| Problem | AWS Backup answer |
|---|---|
| Every service backs up differently, and nobody knows what is covered | One plan and tag policy; AWS Backup Audit Manager reports coverage |
| An attacker or admin deletes backups | Vault access policy, Vault Lock, copies to a separate account |
| Region loss | Cross-Region copy rules |
| "Do the backups restore?" is unknown | Restore testing with validation and timing |
| Compliance wants retention of 7 years | Lifecycle with cold storage and minimum retention enforced by Vault Lock |
Honest limits:
- Not every service is equal. Features such as continuous backup (point-in-time recovery), cold storage and cross-account copy differ by resource type. Check the feature availability table in the AWS Backup documentation before you promise a capability ⚠ VERIFY per resource.
- Backups are not instant. A full backup of a large database takes time; the RPO is the backup interval unless you use continuous backup.
- Vault Lock compliance mode cannot be undone after its grace time. In a lab it can leave you paying for data for years.
- Restore is slow for big data. RTO for a multi-terabyte database is hours; measure it.
Vault Lock in detail
- Governance mode: the lock can be removed by users with sufficient IAM permissions. Good for rehearsing and for teams that want a guard against accidents.
- Compliance mode: after a grace time of at least 3 days (
ChangeableForDays, 3 to 36,500) the vault and its lock become immutable. Nobody, including AWS, can delete the lock or the vault while it contains recovery points. During the grace time you can still change or remove the lock. - Minimum and maximum retention (optional): new backup and copy jobs whose lifecycle is shorter than the minimum, or longer than the maximum, fail. Existing recovery points are unaffected. Minimum retention is at least 1 day, maximum up to 36,500 days. For banking data with a 7-year retention rule, minimum 2,557 days (leap years) is the usual calculation. Your legal team sets the real number.
- Never set retention to "Always" in a locked vault unless you really mean forever: those recovery points cannot be deleted and keep costing money.
- Closing an AWS account suspends it for 90 days and then deletes vault contents, even if locked. Locking does not protect against account closure, so protect the management account too.
There is also a logically air-gapped vault type, designed for storing copies that can be shared across accounts and recovered even if the source account is compromised. It is a separate vault type with its own rules ⚠ VERIFY current features, Region availability and pricing before relying on it.
Cross-account and cross-Region copies
A copy rule in a plan sends each new recovery point to a destination vault in another Region, another account, or both. The usual banking pattern:
- Production account takes the backup into a local vault (fast restore, short retention).
- A copy rule sends it to a central backup account in a second Region, in a Vault Locked vault with the longer retention.
- The central account's access policy allows only AWS Backup copy operations in, and a break-glass role out.
With AWS Organizations you can also manage backup plans centrally using backup policies and enable cross-account management ⚠ VERIFY current setup steps. Encryption matters: a recovery point is encrypted with the vault's KMS key, and for cross-account copies the destination key policy must allow the source. See Backup security and key rotation.
Restore testing
Restore testing is an AWS Backup feature that periodically restores a recovery point, measures the restore time, optionally runs your validation, then deletes the restored resource. The flow:
- You create a restore testing plan: a schedule, a start window (1 to 168 hours), and a recovery point selection (latest or random recovery point within 1 to 365 days, from chosen vaults).
- You add restore testing selections per resource type (DynamoDB, RDS, Aurora, EBS, EC2, EFS, S3 and others in the supported list ⚠ VERIFY), each naming the protected resources or tag conditions, the restore IAM role and the validation window (1 to 168 hours).
- At run time AWS Backup restores, infers sensible restore parameters from the recovery point metadata, and emits a Restore Job State Change event to EventBridge.
- Your validation code (a Lambda) checks the restored resource and calls
PutRestoreValidationResult. - When validation completes or the window closes, AWS Backup deletes the restored resource. Resources are tagged
awsbackup-restore-testwhere tag-on-restore is supported. DynamoDB and S3 restores do not support tag-on-restore, so deletion is by resource name. If you remove that tag from a restored resource, AWS Backup cannot delete it and you must. - AWS Backup Audit Manager has a control that checks whether restore times meet a target, giving you an auditable pass or fail.
Quotas to remember: 100 restore testing plans per account, 30 selections per plan, plan and selection names up to 50 characters.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Backup with Recovery Services vault | AWS Backup with a backup vault | One vault type per account and Region; plans reference vaults by name |
| Backup policy | Backup plan rules and lifecycle | Cold storage and expiry are set per rule |
| Immutable vault, soft delete, multi-user authorisation | Vault Lock, vault access policy, cross-account copy | Vault Lock compliance mode has a grace time and then cannot be changed by anyone |
| Cross-subscription or cross-region restore | Cross-account and cross-Region copy rules | Copy is a rule in the plan, not a restore-time option |
| Backup Center and reports | AWS Backup Audit Manager | Frameworks of controls, with reports delivered to S3 |
SQL Server RESTORE VERIFYONLY plus test restores | Restore testing with a validation Lambda | Automated, scheduled, and measured |
Architecture
Security-boundary view: the production account can write recovery points into the central vault but cannot delete or alter them. The central account is accessed only through a break-glass role. Restore tests run in a dedicated test account or at least a dedicated, isolated VPC and role, because a restore creates real resources that contain real data. Treat a restored production-like table as production data: apply the same access control and encryption.
Step-by-step implementation
Decide what to protect and for how long
Create a table of resources, RPO, retention and copy requirements. Apply a tag such as Backup=daily-35 to every resource that should follow the plan. Tag-based selection means new resources are protected automatically when they carry the tag.
Create the vaults
A local vault with a customer managed KMS key, and a destination vault in the DR Region (and, in a real design, another account). Add a vault access policy that denies backup:DeleteRecoveryPoint to everyone except a break-glass role.
Write the plan
Rule: daily at 03:00 UTC, start window 1 hour, completion window 3 hours, delete after 35 days, copy to the DR vault with a longer retention. Enable continuous backup where supported if your RPO is shorter than a day.
Attach resources by tag
Create a selection with StringEquals on Backup=daily-35 and a role that AWS Backup can assume.
Add Vault Lock (governance in the lab)
Add minimum and maximum retention. In the lab use governance mode so you can remove it. In production consider compliance mode only after legal sign-off and a long grace time.
Add restore testing and validation
Create a testing plan and a selection, then deploy the validation Lambda that subscribes to Restore Job State Change events.
Record the evidence
Export restore job durations and validation outcomes monthly. Compare them with the RTO for the service and keep the report.
AWS Console instructions
- Vault: Console → AWS Backup → Vaults → Create backup vault, choose a KMS key, add tags.
- Plan: Backup plans → Create backup plan → Build a new plan → add a rule (frequency, window, lifecycle) → under Generate copy to destination choose the DR Region vault.
- Assign resources: open the plan → Assign resources → IAM role Default role → Use tags → add key
Backup, valuedaily-35. - Vault Lock: Vaults → select the vault → Vault lock (or Backup vault → Vault locks) → Create vault lock. Choose governance or compliance; compliance asks for the grace time (minimum 3 days) and requires you to type
confirm. - Restore testing: Restore testing → Create restore testing plan → set frequency, start time, start within, source vaults, eligible recovery points and selection criteria → then Assign resources with a restore role and the retention period before cleanup (validation window). Validation itself cannot be started from the console; it is done by API.
AWS CLI
# Create a vault with a customer managed KMS key.
aws backup create-backup-vault --backup-vault-name hl-m12-local \
--encryption-key-arn arn:aws:kms:us-east-1:111122223333:key/REPLACE-ME \
--backup-vault-tags Project=cea-lab,Lab=m12-backup,Environment=learn \
--region us-east-1 --profile learn-sandbox
# DR vault in the second Region.
aws backup create-backup-vault --backup-vault-name hl-m12-dr \
--backup-vault-tags Project=cea-lab,Lab=m12-backup,Environment=learn \
--region us-west-2 --profile learn-sandbox
# Create a plan from a JSON file (see below).
aws backup create-backup-plan --backup-plan file://plan.json \
--region us-east-1 --profile learn-sandbox
# Select resources by tag. Role must be assumable by backup.amazonaws.com.
aws backup create-backup-selection --backup-plan-id PLAN_ID \
--backup-selection file://selection.json --region us-east-1 --profile learn-sandbox
# Take an on-demand backup right now (useful in a lab).
aws backup start-backup-job --backup-vault-name hl-m12-local \
--resource-arn arn:aws:dynamodb:us-east-1:111122223333:table/hl-m12-accounts \
--iam-role-arn arn:aws:iam::111122223333:role/hl-m12-backup-role \
--lifecycle DeleteAfterDays=7 --region us-east-1 --profile learn-sandbox
# Governance-mode Vault Lock: omit --changeable-for-days.
# Adding --changeable-for-days makes it compliance mode (irreversible after the grace time).
aws backup put-backup-vault-lock-configuration --backup-vault-name hl-m12-local \
--min-retention-days 7 --max-retention-days 400 \
--region us-east-1 --profile learn-sandbox
# Check lock state.
aws backup describe-backup-vault --backup-vault-name hl-m12-local \
--query "{Locked:Locked,Min:MinRetentionDays,Max:MaxRetentionDays,Recoveries:NumberOfRecoveryPoints}" \
--region us-east-1 --profile learn-sandbox
# Restore testing plan and selection.
aws backup create-restore-testing-plan --restore-testing-plan file://rtplan.json \
--region us-east-1 --profile learn-sandbox
aws backup create-restore-testing-selection --restore-testing-plan-name hl_m12_weekly \
--restore-testing-selection file://rtselection.json --region us-east-1 --profile learn-sandbox
# List restore jobs and their durations (evidence).
aws backup list-restore-jobs --by-status COMPLETED \
--query "RestoreJobs[].{Id:RestoreJobId,Type:ResourceType,Mins:ExecutionTime,Validation:ValidationStatus}" \
--region us-east-1 --profile learn-sandboxA minimal plan.json:
{
"BackupPlanName": "hl-m12-daily",
"Rules": [
{
"RuleName": "daily-0300-utc",
"TargetBackupVaultName": "hl-m12-local",
"ScheduleExpression": "cron(0 3 * * ? *)",
"StartWindowMinutes": 60,
"CompletionWindowMinutes": 180,
"Lifecycle": { "DeleteAfterDays": 35 },
"CopyActions": [
{
"DestinationBackupVaultArn": "arn:aws:backup:us-west-2:111122223333:backup-vault:hl-m12-dr",
"Lifecycle": { "DeleteAfterDays": 400 }
}
]
}
]
}selection.json:
{
"SelectionName": "tagged-daily-35",
"IamRoleArn": "arn:aws:iam::111122223333:role/hl-m12-backup-role",
"ListOfTags": [
{ "ConditionType": "STRINGEQUALS", "ConditionKey": "Backup", "ConditionValue": "daily-35" }
]
}rtplan.json and rtselection.json:
{
"RestoreTestingPlanName": "hl_m12_weekly",
"ScheduleExpression": "cron(0 6 ? * SUN *)",
"StartWindowHours": 4,
"RecoveryPointSelection": {
"Algorithm": "LATEST_WITHIN_WINDOW",
"IncludeVaults": ["*"],
"RecoveryPointTypes": ["SNAPSHOT"],
"SelectionWindowDays": 8
}
}{
"RestoreTestingSelectionName": "hl_m12_dynamodb",
"ProtectedResourceType": "DynamoDB",
"ProtectedResourceArns": ["*"],
"ProtectedResourceConditions": {
"StringEquals": [{ "Key": "aws:ResourceTag/Backup", "Value": "daily-35" }]
},
"IamRoleArn": "arn:aws:iam::111122223333:role/hl-m12-backup-role",
"ValidationWindowHours": 1
}The wrapper keys of the two restore testing files (RestoreTestingPlan, RestoreTestingSelection) are added by the CLI from the flags, so the files hold the inner object. Check the shape with aws backup create-restore-testing-plan help ⚠ VERIFY. The tag-condition key format (aws:ResourceTag/...) should also be confirmed in the CLI reference ⚠ VERIFY. The selection window must exceed your backup interval plus the start window, as the documentation advises (8 days for a daily backup plus a 4-hour window is generous).
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.Backup;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Constructs;
namespace Harbourline.Ha.Infra;
public sealed class BackupStack : Stack
{
public BackupStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var key = new Key(this, "BackupKey", new KeyProps { EnableKeyRotation = true });
// Governance-mode lock for the lab. Adding GracePeriod would make it compliance mode. ⚠ VERIFY
var vault = new BackupVault(this, "LocalVault", new BackupVaultProps
{
BackupVaultName = "hl-m12-local",
EncryptionKey = key,
LockConfiguration = new LockConfiguration
{
MinRetention = Duration.Days(7),
MaxRetention = Duration.Days(400)
},
RemovalPolicy = RemovalPolicy.DESTROY
});
// Destination vault already created in the DR Region (by another stack).
var drVault = BackupVault.FromBackupVaultArn(this, "DrVault",
$"arn:aws:backup:us-west-2:{Account}:backup-vault:hl-m12-dr");
var table = new Table(this, "Accounts", new TableProps
{
TableName = "hl-m12-accounts",
PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
BillingMode = BillingMode.PAY_PER_REQUEST,
PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification
{
PointInTimeRecoveryEnabled = true
},
RemovalPolicy = RemovalPolicy.DESTROY
});
Amazon.CDK.Tags.Of(table).Add("Backup", "daily-35");
var plan = new BackupPlan(this, "Plan", new BackupPlanProps
{
BackupPlanName = "hl-m12-daily",
BackupVault = vault,
BackupPlanRules = new[]
{
new BackupPlanRule(new BackupPlanRuleProps
{
RuleName = "daily-0300-utc",
ScheduleExpression = Schedule.Cron(new CronOptions { Hour = "3", Minute = "0" }),
StartWindow = Duration.Hours(1),
CompletionWindow = Duration.Hours(3),
DeleteAfter = Duration.Days(35),
CopyActions = new[]
{
new BackupPlanCopyActionProps
{
DestinationBackupVault = drVault,
DeleteAfter = Duration.Days(400)
}
}
})
}
});
plan.AddSelection("TaggedResources", new BackupSelectionOptions
{
Resources = new[] { BackupResource.FromDynamoDbTable(table) }
});
// --- Restore testing ---
var restoreRole = new Role(this, "RestoreRole", new RoleProps
{
AssumedBy = new ServicePrincipal("backup.amazonaws.com"),
ManagedPolicies = new[]
{
ManagedPolicy.FromAwsManagedPolicyName(
"service-role/AWSBackupServiceRolePolicyForRestores")
}
});
var testPlan = new CfnRestoreTestingPlan(this, "RestoreTestPlan", new CfnRestoreTestingPlanProps
{
RestoreTestingPlanName = "hl_m12_weekly",
ScheduleExpression = "cron(0 6 ? * SUN *)",
StartWindowHours = 4,
RecoveryPointSelection = new CfnRestoreTestingPlan.RestoreTestingRecoveryPointSelectionProperty
{
Algorithm = "LATEST_WITHIN_WINDOW",
IncludeVaults = new[] { "*" },
RecoveryPointTypes = new[] { "SNAPSHOT" },
SelectionWindowDays = 8
}
});
_ = new CfnRestoreTestingSelection(this, "RestoreTestSelection", new CfnRestoreTestingSelectionProps
{
RestoreTestingPlanName = testPlan.RestoreTestingPlanName,
RestoreTestingSelectionName = "hl_m12_dynamodb",
ProtectedResourceType = "DynamoDB",
ProtectedResourceArns = new[] { table.TableArn },
IamRoleArn = restoreRole.RoleArn,
ValidationWindowHours = 1
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m12-backup");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
}
}CfnRestoreTestingSelection has a RestoreTestingPlanName property that expects the plan name. If CDK reports a type mismatch in your version, pass the literal string "hl_m12_weekly" and add AddDependency ⚠ VERIFY. The restore testing Lambda is added in the next section.
Complete code
The validation Lambda is the part that turns "restore completed" into "restore proved the data is there". It reads the restored DynamoDB table, compares item count and a sample against expectations, and reports to AWS Backup.
using Amazon.Backup;
using Amazon.Backup.Model;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.Core;
using System.Text.Json;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace Harbourline.Ha.RestoreValidator;
public sealed class Function(IAmazonBackup backup, IAmazonDynamoDB dynamo)
{
// Lambda runtime constructor.
public Function() : this(new AmazonBackupClient(), new AmazonDynamoDBClient()) { }
public async Task Handler(JsonElement evt, ILambdaContext ctx)
{
var detail = evt.GetProperty("detail");
var status = detail.GetProperty("status").GetString();
if (status != "COMPLETED") return; // only validate finished restores
var jobId = detail.GetProperty("restoreJobId").GetString()!;
var createdArn = detail.GetProperty("createdResourceArn").GetString()!;
var resourceType = detail.GetProperty("resourceType").GetString();
if (resourceType != "DynamoDB")
{
ctx.Logger.LogInformation($"Skipping {resourceType}: no validator");
return;
}
var tableName = createdArn.Split("table/")[1];
var (ok, message) = await ValidateAsync(tableName);
await backup.PutRestoreValidationResultAsync(new PutRestoreValidationResultRequest
{
RestoreJobId = jobId,
ValidationStatus = ok ? RestoreValidationStatus.SUCCESSFUL : RestoreValidationStatus.FAILED,
ValidationStatusMessage = message
});
ctx.Logger.LogInformation($"Restore {jobId}: {(ok ? "SUCCESSFUL" : "FAILED")} - {message}");
}
internal async Task<(bool Ok, string Message)> ValidateAsync(string tableName)
{
var desc = await dynamo.DescribeTableAsync(tableName);
if (desc.Table.TableStatus != TableStatus.ACTIVE)
return (false, $"Table status is {desc.Table.TableStatus}, expected ACTIVE");
// Item count in DescribeTable is refreshed about every 6 hours, so scan a bounded sample.
var scan = await dynamo.ScanAsync(new ScanRequest { TableName = tableName, Limit = 25 });
return RestoreRules.Evaluate(scan.Items.Count, scan.Items.Select(i => i.Keys.ToHashSet()));
}
}
public static class RestoreRules
{
private static readonly string[] RequiredAttributes = ["pk", "balance", "currency"];
public static (bool Ok, string Message) Evaluate(int sampleCount, IEnumerable<HashSet<string>> itemAttributeSets)
{
if (sampleCount == 0) return (false, "Restored table is empty");
var bad = itemAttributeSets.Count(keys => !RequiredAttributes.All(keys.Contains));
return bad == 0
? (true, $"Sampled {sampleCount} items; all have required attributes")
: (false, $"{bad} of {sampleCount} sampled items miss required attributes");
}
}A tiny evidence helper compares each restore job's duration with the service's RTO:
namespace Harbourline.Ha.RestoreValidator;
public sealed record RestoreRecord(string JobId, string Resource, TimeSpan Duration, string Validation);
public static class RestoreEvidence
{
/// <summary>Returns a markdown table line per job, flagging jobs slower than the RTO share.</summary>
public static IEnumerable<string> Report(IEnumerable<RestoreRecord> jobs, TimeSpan rto, double rtoShare = 0.5)
{
// A restore is only one part of recovery: allow it half of the RTO by default.
var budget = TimeSpan.FromTicks((long)(rto.Ticks * rtoShare));
foreach (var j in jobs)
{
var flag = j.Duration <= budget && j.Validation == "SUCCESSFUL" ? "PASS" : "REVIEW";
yield return $"| {j.JobId} | {j.Resource} | {j.Duration.TotalMinutes:F1} min | {j.Validation} | {flag} |";
}
}
}An EventBridge rule (CDK C#) connects the event to the function:
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.Lambda;
using Constructs;
namespace Harbourline.Ha.Infra;
public static class RestoreValidationRule
{
public static void Add(Construct scope, IFunction validator)
{
_ = new Rule(scope, "RestoreCompleted", new RuleProps
{
EventPattern = new EventPattern
{
Source = new[] { "aws.backup" },
DetailType = new[] { "Restore Job State Change" },
Detail = new Dictionary<string, object>
{
["status"] = new[] { "COMPLETED" }
}
},
Targets = new IRuleTarget[] { new LambdaFunction(validator) }
});
}
}The Lambda role needs backup:PutRestoreValidationResult, dynamodb:DescribeTable and dynamodb:Scan on arn:aws:dynamodb:*:*:table/awsbackup-restore-test* ⚠ VERIFY the restored table's naming. Put the code in labs/ and unit-test RestoreRules and RestoreEvidence with xUnit v3 without AWS.
Code explained line by line
Function(IAmazonBackup backup, IAmazonDynamoDB dynamo)uses a primary constructor so tests can pass fakes. The parameterless constructor is what the Lambda runtime calls.- The handler ignores events that are not
COMPLETED; AWS Backup emits state changes for every transition. createdResourceArnidentifies the restored table. A restore test names the new table itself, so you never guess; split ontable/.ValidateAsyncchecks that the table isACTIVEbefore looking inside. A restored table that is still creating would give a false failure.- The scan is bounded with
Limit = 25. A full scan on a large table would be slow and expensive. Item counts fromDescribeTableare updated only periodically, so they are poor evidence. RestoreRules.Evaluateis a pure function that checks required attributes. Real validation would compare a checksum of a known sample (thefingerprintidea from Module 6), a recent known transaction, or the highestbooking_date.PutRestoreValidationResultAsyncreportsSUCCESSFULorFAILED, with a short message. If you do not report within the validation window, AWS Backup deletes the restored resource anyway.RestoreEvidence.Reportbudgets half of the RTO for the restore step, because failover, DNS and validation also take time.
Security considerations
⚠ Learning shortcut — not for production
The lab uses governance-mode Vault Lock, one account, default keys policy and RemovalPolicy.DESTROY. Production banks use a separate backup account, a Vault Locked vault whose retention comes from the records-retention policy, a KMS key policy that grants only AWS Backup and a break-glass role, and an SCP that denies backup:DeleteBackupVault, backup:DeleteRecoveryPoint and backup:PutBackupVaultLockConfiguration outside the break-glass role.
- Compliance mode is a one-way door. Use a throwaway account, a minimum 3-day grace time and a very small minimum retention for any experiment, or avoid it in a lab entirely. Do not lock a vault in a shared sandbox.
- Restore creates real resources with real data. Restore tests should run in an isolated account or at least with tight IAM, no internet path, and the same encryption as production. Restored tables are deleted automatically after the validation window.
- KMS. Cross-account copy requires the destination vault key to allow the source account. A common failure is a copy job stuck on
AccessDeniedfor KMS. - Least privilege on the restore role. Use the AWS-managed restore policy as a starting point, then scope down.
- Audit. Turn on AWS Backup Audit Manager frameworks for "resources are in a backup plan", "backup plan min frequency", "cross-Region copy" and the restore-time control, and keep reports in an S3 bucket with Object Lock. See S3 protection and Object Lock.
- UK GDPR and erasure. An immutable backup holds personal data until retention ends. Write down how erasure requests are handled for backed-up data (typically "put beyond use" and expire on schedule). Ask your data-protection officer.
Banking use case
Harbourline's backup policy for important business services:
| Service data | Source | Local retention | Copy | Lock |
|---|---|---|---|---|
| Ledger database | Aurora | 35 days plus continuous backup | DR Region, central backup account | Compliance, minimum 2,557 days for monthly recovery points ⚠ legal to confirm |
| Account index | DynamoDB | 35 days | DR Region | Governance |
| Customer documents | S3 | Versioning plus Backup | Another account | Compliance, per records policy |
| Build artefacts | ECR, S3 | 7 days | None | None |
Every quarter, the operational-resilience team runs the restore testing report: restore time, validation result and comparison with the service's RTO. A failed validation raises an incident ticket automatically through an EventBridge rule. That report is a core exhibit for the regulator conversation. Harbourline does not claim the report equals compliance; it is one piece of evidence the compliance team uses.
Common mistakes
- Backups in the same account and Region as production. One compromised credential or one Regional event deletes everything. Always copy out.
- Compliance-mode lock set in a hurry. The grace time is the only undo. Review minimum and maximum retention with legal first and never use "Always" retention in locked vaults.
- Assuming "backup succeeded" means "restore works". Missing KMS permissions, wrong subnet groups and default VPC assumptions break restores. Restore testing finds these while you are calm.
- Forgetting the tag in resource selection. A new table without the
Backuptag is silently unprotected. Add an AWS Config rule or Audit Manager control for coverage. - Retention shorter than detection time. If it takes 40 days to notice quiet corruption, 35-day retention is useless. Keep monthly recovery points longer.
- Expecting restore testing to touch everything. It supports a list of resource types; confirm yours is included ⚠ VERIFY.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Backup job FAILED: access denied | Role cannot be assumed by backup.amazonaws.com or lacks service permission | Fix the trust policy and attach the AWS-managed backup policy |
| Copy job fails with KMS error | Destination key policy does not allow the source, or source key not shared | Update key policies on both sides |
| New backup job rejected by vault | Lifecycle is shorter than vault lock minimum or longer than maximum | Align the plan's DeleteAfter with lock min and max |
Restore test FAILED: no default VPC or subnet | EC2 or RDS restore needs a subnet and the account has no default VPC | Override SubnetId or DBSubnetGroupName in the restore testing selection |
| Restored resource not deleted | awsbackup-restore-test tag removed, or deletion still in progress | Delete it manually, check the deletion status with describe-restore-job |
| Validation never reported | Lambda did not receive the event, or the window closed | Check the EventBridge rule pattern and the validation window hours |
delete-backup-vault fails | Vault still contains recovery points or is locked | Delete recovery points (allowed in governance mode with permissions) or wait for retention |
| Restore testing selection rejected | Both ARNs and conditions supplied | Use specific ARNs or wildcard plus conditions, not both |
Cost and cleanup
Cost: credits. Backup storage is billed per GB-month; a tiny DynamoDB table costs cents. Cross-Region copy adds data transfer. Restore tests carry a per-test charge plus the cost of the restored resource while it exists ⚠ VERIFY on the AWS Backup pricing page. Keep the lab to one tiny table, one backup, one restore test, and delete within the day. Free alternative: design the plan, run the C# validator rules in unit tests, and read the AWS Backup documentation tables; no cloud account needed.
# 1. Remove the Vault Lock first if you used governance mode (compliance mode cannot be removed after grace time).
aws backup delete-backup-vault-lock-configuration --backup-vault-name hl-m12-local \
--region us-east-1 --profile learn-sandbox
# 2. Delete restore testing selections, then the plan.
aws backup delete-restore-testing-selection --restore-testing-plan-name hl_m12_weekly \
--restore-testing-selection-name hl_m12_dynamodb --region us-east-1 --profile learn-sandbox
aws backup delete-restore-testing-plan --restore-testing-plan-name hl_m12_weekly \
--region us-east-1 --profile learn-sandbox
# 3. Delete backup selection and plan.
aws backup list-backup-plans --region us-east-1 --profile learn-sandbox
aws backup list-backup-selections --backup-plan-id PLAN_ID --region us-east-1 --profile learn-sandbox
aws backup delete-backup-selection --backup-plan-id PLAN_ID --selection-id SELECTION_ID \
--region us-east-1 --profile learn-sandbox
aws backup delete-backup-plan --backup-plan-id PLAN_ID --region us-east-1 --profile learn-sandbox
# 4. Delete every recovery point in both vaults, then the vaults.
aws backup list-recovery-points-by-backup-vault --backup-vault-name hl-m12-local \
--query "RecoveryPoints[].RecoveryPointArn" --output text --region us-east-1 --profile learn-sandbox
aws backup delete-recovery-point --backup-vault-name hl-m12-local \
--recovery-point-arn RECOVERY_POINT_ARN --region us-east-1 --profile learn-sandbox
aws backup delete-backup-vault --backup-vault-name hl-m12-local --region us-east-1 --profile learn-sandbox
aws backup delete-backup-vault --backup-vault-name hl-m12-dr --region us-west-2 --profile learn-sandbox
# 5. Delete the DynamoDB table and the KMS key (schedule deletion), then verify.
aws dynamodb delete-table --table-name hl-m12-accounts --region us-east-1 --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"Repeat step 4's recovery point deletion for hl-m12-dr in us-west-2. If you used CDK or Terraform, cdk destroy or terraform destroy removes the stack, but a vault that still contains recovery points will not delete, so empty it first.
Exercises
- 🟢 Guided: Create the vault, plan and selection with the CLI for one tagged DynamoDB table, run
start-backup-job, and note the time the recovery point appears. Then restore it to a new table withstart-restore-joband compare item counts. - 🟡 Challenge: Add restore testing for the table and a validation Lambda. Write unit tests for
RestoreRules.Evaluatecovering an empty table, a table with a missing attribute and a good table. - 🔴 Live problem: The compliance team says "we need immutable backups for 7 years, but last quarter's restore test took 3 hours and our RTO is 2 hours". Propose changes (continuous backup, warm standby, restore parallelism, smaller recovery scope) and explain what evidence you would show next quarter.
Assignment
Write a backup and restore standard for one Harbourline service: resources and tags, plan rules, local and copy retention, lock mode and its approval, restore testing frequency, validation checks, and the measured-versus-target RTO table. Include the IaC. Commit it to your portfolio repository.
Solutions
aws backup start-backup-job returns a BackupJobId; poll with describe-backup-job until COMPLETED, then list-recovery-points-by-backup-vault. Restore with aws backup start-restore-job --recovery-point-arn ... --iam-role-arn ... --resource-type DynamoDB --metadata targetTableName=hl-m12-accounts-restored ⚠ VERIFY the exact metadata keys with get-restore-job-metadata. Compare counts with aws dynamodb scan --select COUNT. The point is to feel the restore time yourself: that number goes in the RTO table. Delete the restored table afterwards.
Create the plan and selection (from the IaC), deploy the Lambda and EventBridge rule, then start a restore test manually or wait for the schedule. Unit tests for RestoreRules.Evaluate: Evaluate(0, []) returns (false, "Restored table is empty"); one item whose key set lacks balance gives a failure message mentioning 1 of 1; a set containing pk, balance and currency gives success. Pure functions make validation reviewable and cheap to test.
Knowledge check
Answer: Governance mode can be removed by users with the right IAM permissions. Compliance mode has a grace time (at least 3 days) after which the lock cannot be changed or removed by anyone, including AWS, while the vault has recovery points. Compliance mode is for regulatory immutability and must be chosen deliberately.
Answer: Another Region protects against a Regional event; another account protects against compromise of the production account's credentials. An attacker with admin rights in production can delete same-account backups, but not those in an account they cannot reach.
Answer: It schedules restores, measures duration, deletes the restored resource and records results. The business-level check that the data is correct is your validation code, reported with PutRestoreValidationResult within the validation window.
Answer: New backup and copy jobs must have a lifecycle within the vault's minimum and maximum retention. A 7-day lifecycle is below the 30-day minimum, so the job fails. Change the plan's DeleteAfter or the lock.
Answer: No. Multi-AZ replicates every change, including a destructive one, within milliseconds. Backups are point-in-time copies that let you return to before the damage.
Interview questions
- How would you protect backups from a ransomware actor with admin access to production? Cross-account copy to a locked vault, SCPs denying deletion, KMS keys in the backup account, MFA-protected break-glass access, and monitored
DeleteRecoveryPointattempts. - How do you prove a backup is usable? Automated restore testing with validation, measured duration compared with the RTO, and a retained report.
- What are the trade-offs of continuous backup? Lower RPO (point-in-time restore within a retention window) at higher storage cost and not available for every resource ⚠ VERIFY; restores still take time.
- Explain the risk of compliance-mode Vault Lock. After the grace time it cannot be undone, so mistaken retention values keep costing money and keep data you may have to delete.
Certification practice
A company must ensure that backups of an RDS database cannot be deleted by any user, including the root user, for 7 years, and that a mistaken configuration can be corrected for three days after setup. Which configuration meets this?
A. A backup vault with a vault access policy that denies deletion. B. A backup vault with Vault Lock in compliance mode and a grace time of 3 days, with minimum retention set to 7 years. C. S3 Object Lock on a bucket holding RDS snapshot exports. D. Governance-mode Vault Lock.
Answer: B. Compliance mode becomes immutable after the grace time (minimum 3 days), and minimum retention enforces the period. A can be changed by anyone with policy permissions. C protects exports, not the Backup recovery points. D can be removed by authorised users.
A team wants backups of DynamoDB and EBS volumes managed by one policy, with copies in another Region, selected by tag. Which service?
A. Amazon Data Lifecycle Manager only.
B. AWS Backup with a backup plan, tag-based selection and a copy rule.
C. Custom Lambda functions calling CreateBackup per service.
D. AWS Elastic Disaster Recovery.
Answer: B. AWS Backup centralises many services with tag selection and copy actions. A covers EBS snapshots but not DynamoDB. C works but reimplements the service. D replicates servers for recovery, not centralised backup.
You must automatically verify each week that the latest recovery point of an Aurora cluster restores and that a sample query returns expected rows, then remove the test cluster. What do you configure?
A. A CloudWatch alarm on backup job failures.
B. AWS Backup restore testing with a validation window and a Lambda that calls PutRestoreValidationResult.
C. A Step Functions workflow that runs restore-db-cluster-from-snapshot and never deletes the result.
D. Cross-Region copy of the recovery point.
Answer: B. Restore testing restores on a schedule, supports validation, and deletes the restored resource. A only checks backups, not restores. C leaves resources behind and reinvents the feature. D does not test restores.
Completion checklist
- I can explain vault, recovery point, plan, selection, Vault Lock and restore testing
- I know why compliance-mode Vault Lock must be handled carefully
- I completed the guided exercise and measured one restore time
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- AWS Backup gives one policy for many services: plans, tag selection, lifecycle and copies.
- Immutable backups need separation: another account, another Region, a locked vault.
- A backup is a hope until restore testing proves it, with validation and a timing.
- Compliance-mode Vault Lock is irreversible after the grace time: plan retention with legal first.
- Keep evidence: restore duration against RTO, validation status, and the audit report.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Database failover across Regions
Choose and rehearse cross-Region database recovery: Aurora Global Database switchover and failover, DynamoDB global tables (MREC and MRSC), and RDS cross-Region read replica promotion, with a runbook and C# helpers that make the decision measurable.
Caching for performance and protection
Use CloudFront and ElastiCache to speed up reads and shield back ends from load spikes and partial outages, implement cache-aside with HybridCache in .NET, and apply a data-classification rule for what must never be cached.