Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

AWS Backup in depth

Build a backup plan with vaults, Vault Lock, cross-Region and cross-account copies, and automated restore testing with validation, then produce restore evidence a bank can show an auditor.

intermediate⏱ 2 h 30 min🟡 Low costSAA-C03SOA-C03DOP-C02SAP-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Backup, AWSSDK.DynamoDBv2) · AWS CLI v2 · AWS CDK v2 (C#) · Terraform AWS provider 6.x · Region us-east-1 (copy target us-west-2). Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Build an AWS Backup plan with schedule, lifecycle, tag-based resource selection and a cross-Region copy.
  • Explain vault access policies, Vault Lock (governance and compliance mode) and why compliance mode is irreversible after its grace time.
  • Design a cross-account backup so that a compromised production account cannot delete its own backups.
  • Configure restore testing with a validation step, and write a C# Lambda that validates a restored DynamoDB table and reports the result.
  • Turn restore job timings into evidence against an RTO.

Prerequisites

Why this matters in banking

A bank's worst day is not always a dead data centre. It can be a ransomware actor who has gained administrator rights in the production account, found the backups, and deleted them before encrypting the data. Replication does not help (it copies the damage) and neither do backups that the same administrator can delete.

The control that matters is immutable, separated backups: copies in a different account, in a vault that even the account's administrators cannot empty before the retention period ends, plus proof that you can restore from them. Operational-resilience regimes ask for exactly that kind of evidence: not "we have backups", but "we restored last month's backup in 22 minutes and the data checked out".

The idea in plain language

AWS Backup is a central service that takes scheduled backups of many different AWS resources using one policy language. Instead of each team configuring RDS snapshots, DynamoDB backups, EBS snapshots and EFS backups differently, you write a plan: when to back up, how long to keep, where to copy. Resources are attached to the plan by tags or ARNs.

The vocabulary:

  • Backup vault: a container for backups (recovery points), encrypted with a KMS key, with an access policy.
  • Recovery point: one backup of one resource.
  • Backup plan: rules (schedule, window, lifecycle, copy actions).
  • Selection (resource assignment): which resources the plan covers, usually by tag.
  • Vault Lock: a write-once-read-many (WORM) setting that stops anyone deleting recovery points before their retention ends.
  • Restore testing: scheduled automatic restores of recent recovery points, with optional validation, measuring how long each takes.

Analogy: The backup plan is the bank's cash-in-transit schedule, a vault is the secure room it goes to, and Vault Lock is a time lock that nobody, including the branch manager, can open early. Restore testing is the monthly audit where you really open a box and count the notes.

What problem AWS Backup solves

ProblemAWS Backup answer
Every service backs up differently, and nobody knows what is coveredOne plan and tag policy; AWS Backup Audit Manager reports coverage
An attacker or admin deletes backupsVault access policy, Vault Lock, copies to a separate account
Region lossCross-Region copy rules
"Do the backups restore?" is unknownRestore testing with validation and timing
Compliance wants retention of 7 yearsLifecycle with cold storage and minimum retention enforced by Vault Lock

Honest limits:

  • Not every service is equal. Features such as continuous backup (point-in-time recovery), cold storage and cross-account copy differ by resource type. Check the feature availability table in the AWS Backup documentation before you promise a capability ⚠ VERIFY per resource.
  • Backups are not instant. A full backup of a large database takes time; the RPO is the backup interval unless you use continuous backup.
  • Vault Lock compliance mode cannot be undone after its grace time. In a lab it can leave you paying for data for years.
  • Restore is slow for big data. RTO for a multi-terabyte database is hours; measure it.

Vault Lock in detail

  • Governance mode: the lock can be removed by users with sufficient IAM permissions. Good for rehearsing and for teams that want a guard against accidents.
  • Compliance mode: after a grace time of at least 3 days (ChangeableForDays, 3 to 36,500) the vault and its lock become immutable. Nobody, including AWS, can delete the lock or the vault while it contains recovery points. During the grace time you can still change or remove the lock.
  • Minimum and maximum retention (optional): new backup and copy jobs whose lifecycle is shorter than the minimum, or longer than the maximum, fail. Existing recovery points are unaffected. Minimum retention is at least 1 day, maximum up to 36,500 days. For banking data with a 7-year retention rule, minimum 2,557 days (leap years) is the usual calculation. Your legal team sets the real number.
  • Never set retention to "Always" in a locked vault unless you really mean forever: those recovery points cannot be deleted and keep costing money.
  • Closing an AWS account suspends it for 90 days and then deletes vault contents, even if locked. Locking does not protect against account closure, so protect the management account too.

There is also a logically air-gapped vault type, designed for storing copies that can be shared across accounts and recovered even if the source account is compromised. It is a separate vault type with its own rules ⚠ VERIFY current features, Region availability and pricing before relying on it.

Cross-account and cross-Region copies

A copy rule in a plan sends each new recovery point to a destination vault in another Region, another account, or both. The usual banking pattern:

  1. Production account takes the backup into a local vault (fast restore, short retention).
  2. A copy rule sends it to a central backup account in a second Region, in a Vault Locked vault with the longer retention.
  3. The central account's access policy allows only AWS Backup copy operations in, and a break-glass role out.

With AWS Organizations you can also manage backup plans centrally using backup policies and enable cross-account management ⚠ VERIFY current setup steps. Encryption matters: a recovery point is encrypted with the vault's KMS key, and for cross-account copies the destination key policy must allow the source. See Backup security and key rotation.

Restore testing

Restore testing is an AWS Backup feature that periodically restores a recovery point, measures the restore time, optionally runs your validation, then deletes the restored resource. The flow:

  1. You create a restore testing plan: a schedule, a start window (1 to 168 hours), and a recovery point selection (latest or random recovery point within 1 to 365 days, from chosen vaults).
  2. You add restore testing selections per resource type (DynamoDB, RDS, Aurora, EBS, EC2, EFS, S3 and others in the supported list ⚠ VERIFY), each naming the protected resources or tag conditions, the restore IAM role and the validation window (1 to 168 hours).
  3. At run time AWS Backup restores, infers sensible restore parameters from the recovery point metadata, and emits a Restore Job State Change event to EventBridge.
  4. Your validation code (a Lambda) checks the restored resource and calls PutRestoreValidationResult.
  5. When validation completes or the window closes, AWS Backup deletes the restored resource. Resources are tagged awsbackup-restore-test where tag-on-restore is supported. DynamoDB and S3 restores do not support tag-on-restore, so deletion is by resource name. If you remove that tag from a restored resource, AWS Backup cannot delete it and you must.
  6. AWS Backup Audit Manager has a control that checks whether restore times meet a target, giving you an auditable pass or fail.

Quotas to remember: 100 restore testing plans per account, 30 selections per plan, plan and selection names up to 50 characters.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Backup with Recovery Services vaultAWS Backup with a backup vaultOne vault type per account and Region; plans reference vaults by name
Backup policyBackup plan rules and lifecycleCold storage and expiry are set per rule
Immutable vault, soft delete, multi-user authorisationVault Lock, vault access policy, cross-account copyVault Lock compliance mode has a grace time and then cannot be changed by anyone
Cross-subscription or cross-region restoreCross-account and cross-Region copy rulesCopy is a rule in the plan, not a restore-time option
Backup Center and reportsAWS Backup Audit ManagerFrameworks of controls, with reports delivered to S3
SQL Server RESTORE VERIFYONLY plus test restoresRestore testing with a validation LambdaAutomated, scheduled, and measured

Architecture

Security-boundary view: the production account can write recovery points into the central vault but cannot delete or alter them. The central account is accessed only through a break-glass role. Restore tests run in a dedicated test account or at least a dedicated, isolated VPC and role, because a restore creates real resources that contain real data. Treat a restored production-like table as production data: apply the same access control and encryption.

Step-by-step implementation

Decide what to protect and for how long

Create a table of resources, RPO, retention and copy requirements. Apply a tag such as Backup=daily-35 to every resource that should follow the plan. Tag-based selection means new resources are protected automatically when they carry the tag.

Create the vaults

A local vault with a customer managed KMS key, and a destination vault in the DR Region (and, in a real design, another account). Add a vault access policy that denies backup:DeleteRecoveryPoint to everyone except a break-glass role.

Write the plan

Rule: daily at 03:00 UTC, start window 1 hour, completion window 3 hours, delete after 35 days, copy to the DR vault with a longer retention. Enable continuous backup where supported if your RPO is shorter than a day.

Attach resources by tag

Create a selection with StringEquals on Backup=daily-35 and a role that AWS Backup can assume.

Add Vault Lock (governance in the lab)

Add minimum and maximum retention. In the lab use governance mode so you can remove it. In production consider compliance mode only after legal sign-off and a long grace time.

Add restore testing and validation

Create a testing plan and a selection, then deploy the validation Lambda that subscribes to Restore Job State Change events.

Record the evidence

Export restore job durations and validation outcomes monthly. Compare them with the RTO for the service and keep the report.

AWS Console instructions

  • Vault: Console → AWS Backup → Vaults → Create backup vault, choose a KMS key, add tags.
  • Plan: Backup plans → Create backup plan → Build a new plan → add a rule (frequency, window, lifecycle) → under Generate copy to destination choose the DR Region vault.
  • Assign resources: open the plan → Assign resources → IAM role Default role → Use tags → add key Backup, value daily-35.
  • Vault Lock: Vaults → select the vault → Vault lock (or Backup vault → Vault locks) → Create vault lock. Choose governance or compliance; compliance asks for the grace time (minimum 3 days) and requires you to type confirm.
  • Restore testing: Restore testing → Create restore testing plan → set frequency, start time, start within, source vaults, eligible recovery points and selection criteria → then Assign resources with a restore role and the retention period before cleanup (validation window). Validation itself cannot be started from the console; it is done by API.

AWS CLI

# Create a vault with a customer managed KMS key.
aws backup create-backup-vault --backup-vault-name hl-m12-local \
  --encryption-key-arn arn:aws:kms:us-east-1:111122223333:key/REPLACE-ME \
  --backup-vault-tags Project=cea-lab,Lab=m12-backup,Environment=learn \
  --region us-east-1 --profile learn-sandbox

# DR vault in the second Region.
aws backup create-backup-vault --backup-vault-name hl-m12-dr \
  --backup-vault-tags Project=cea-lab,Lab=m12-backup,Environment=learn \
  --region us-west-2 --profile learn-sandbox

# Create a plan from a JSON file (see below).
aws backup create-backup-plan --backup-plan file://plan.json \
  --region us-east-1 --profile learn-sandbox

# Select resources by tag. Role must be assumable by backup.amazonaws.com.
aws backup create-backup-selection --backup-plan-id PLAN_ID \
  --backup-selection file://selection.json --region us-east-1 --profile learn-sandbox

# Take an on-demand backup right now (useful in a lab).
aws backup start-backup-job --backup-vault-name hl-m12-local \
  --resource-arn arn:aws:dynamodb:us-east-1:111122223333:table/hl-m12-accounts \
  --iam-role-arn arn:aws:iam::111122223333:role/hl-m12-backup-role \
  --lifecycle DeleteAfterDays=7 --region us-east-1 --profile learn-sandbox

# Governance-mode Vault Lock: omit --changeable-for-days.
# Adding --changeable-for-days makes it compliance mode (irreversible after the grace time).
aws backup put-backup-vault-lock-configuration --backup-vault-name hl-m12-local \
  --min-retention-days 7 --max-retention-days 400 \
  --region us-east-1 --profile learn-sandbox

# Check lock state.
aws backup describe-backup-vault --backup-vault-name hl-m12-local \
  --query "{Locked:Locked,Min:MinRetentionDays,Max:MaxRetentionDays,Recoveries:NumberOfRecoveryPoints}" \
  --region us-east-1 --profile learn-sandbox

# Restore testing plan and selection.
aws backup create-restore-testing-plan --restore-testing-plan file://rtplan.json \
  --region us-east-1 --profile learn-sandbox
aws backup create-restore-testing-selection --restore-testing-plan-name hl_m12_weekly \
  --restore-testing-selection file://rtselection.json --region us-east-1 --profile learn-sandbox

# List restore jobs and their durations (evidence).
aws backup list-restore-jobs --by-status COMPLETED \
  --query "RestoreJobs[].{Id:RestoreJobId,Type:ResourceType,Mins:ExecutionTime,Validation:ValidationStatus}" \
  --region us-east-1 --profile learn-sandbox

A minimal plan.json:

{
  "BackupPlanName": "hl-m12-daily",
  "Rules": [
    {
      "RuleName": "daily-0300-utc",
      "TargetBackupVaultName": "hl-m12-local",
      "ScheduleExpression": "cron(0 3 * * ? *)",
      "StartWindowMinutes": 60,
      "CompletionWindowMinutes": 180,
      "Lifecycle": { "DeleteAfterDays": 35 },
      "CopyActions": [
        {
          "DestinationBackupVaultArn": "arn:aws:backup:us-west-2:111122223333:backup-vault:hl-m12-dr",
          "Lifecycle": { "DeleteAfterDays": 400 }
        }
      ]
    }
  ]
}

selection.json:

{
  "SelectionName": "tagged-daily-35",
  "IamRoleArn": "arn:aws:iam::111122223333:role/hl-m12-backup-role",
  "ListOfTags": [
    { "ConditionType": "STRINGEQUALS", "ConditionKey": "Backup", "ConditionValue": "daily-35" }
  ]
}

rtplan.json and rtselection.json:

{
  "RestoreTestingPlanName": "hl_m12_weekly",
  "ScheduleExpression": "cron(0 6 ? * SUN *)",
  "StartWindowHours": 4,
  "RecoveryPointSelection": {
    "Algorithm": "LATEST_WITHIN_WINDOW",
    "IncludeVaults": ["*"],
    "RecoveryPointTypes": ["SNAPSHOT"],
    "SelectionWindowDays": 8
  }
}
{
  "RestoreTestingSelectionName": "hl_m12_dynamodb",
  "ProtectedResourceType": "DynamoDB",
  "ProtectedResourceArns": ["*"],
  "ProtectedResourceConditions": {
    "StringEquals": [{ "Key": "aws:ResourceTag/Backup", "Value": "daily-35" }]
  },
  "IamRoleArn": "arn:aws:iam::111122223333:role/hl-m12-backup-role",
  "ValidationWindowHours": 1
}

The wrapper keys of the two restore testing files (RestoreTestingPlan, RestoreTestingSelection) are added by the CLI from the flags, so the files hold the inner object. Check the shape with aws backup create-restore-testing-plan help ⚠ VERIFY. The tag-condition key format (aws:ResourceTag/...) should also be confirmed in the CLI reference ⚠ VERIFY. The selection window must exceed your backup interval plus the start window, as the documentation advises (8 days for a daily backup plus a 4-hour window is generous).

Infrastructure as Code

infra/cdk/BackupStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.Backup;
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Constructs;

namespace Harbourline.Ha.Infra;

public sealed class BackupStack : Stack
{
    public BackupStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var key = new Key(this, "BackupKey", new KeyProps { EnableKeyRotation = true });

        // Governance-mode lock for the lab. Adding GracePeriod would make it compliance mode. ⚠ VERIFY
        var vault = new BackupVault(this, "LocalVault", new BackupVaultProps
        {
            BackupVaultName = "hl-m12-local",
            EncryptionKey = key,
            LockConfiguration = new LockConfiguration
            {
                MinRetention = Duration.Days(7),
                MaxRetention = Duration.Days(400)
            },
            RemovalPolicy = RemovalPolicy.DESTROY
        });

        // Destination vault already created in the DR Region (by another stack).
        var drVault = BackupVault.FromBackupVaultArn(this, "DrVault",
            $"arn:aws:backup:us-west-2:{Account}:backup-vault:hl-m12-dr");

        var table = new Table(this, "Accounts", new TableProps
        {
            TableName = "hl-m12-accounts",
            PartitionKey = new Amazon.CDK.AWS.DynamoDB.Attribute { Name = "pk", Type = AttributeType.STRING },
            BillingMode = BillingMode.PAY_PER_REQUEST,
            PointInTimeRecoverySpecification = new PointInTimeRecoverySpecification
            {
                PointInTimeRecoveryEnabled = true
            },
            RemovalPolicy = RemovalPolicy.DESTROY
        });
        Amazon.CDK.Tags.Of(table).Add("Backup", "daily-35");

        var plan = new BackupPlan(this, "Plan", new BackupPlanProps
        {
            BackupPlanName = "hl-m12-daily",
            BackupVault = vault,
            BackupPlanRules = new[]
            {
                new BackupPlanRule(new BackupPlanRuleProps
                {
                    RuleName = "daily-0300-utc",
                    ScheduleExpression = Schedule.Cron(new CronOptions { Hour = "3", Minute = "0" }),
                    StartWindow = Duration.Hours(1),
                    CompletionWindow = Duration.Hours(3),
                    DeleteAfter = Duration.Days(35),
                    CopyActions = new[]
                    {
                        new BackupPlanCopyActionProps
                        {
                            DestinationBackupVault = drVault,
                            DeleteAfter = Duration.Days(400)
                        }
                    }
                })
            }
        });

        plan.AddSelection("TaggedResources", new BackupSelectionOptions
        {
            Resources = new[] { BackupResource.FromDynamoDbTable(table) }
        });

        // --- Restore testing ---
        var restoreRole = new Role(this, "RestoreRole", new RoleProps
        {
            AssumedBy = new ServicePrincipal("backup.amazonaws.com"),
            ManagedPolicies = new[]
            {
                ManagedPolicy.FromAwsManagedPolicyName(
                    "service-role/AWSBackupServiceRolePolicyForRestores")
            }
        });

        var testPlan = new CfnRestoreTestingPlan(this, "RestoreTestPlan", new CfnRestoreTestingPlanProps
        {
            RestoreTestingPlanName = "hl_m12_weekly",
            ScheduleExpression = "cron(0 6 ? * SUN *)",
            StartWindowHours = 4,
            RecoveryPointSelection = new CfnRestoreTestingPlan.RestoreTestingRecoveryPointSelectionProperty
            {
                Algorithm = "LATEST_WITHIN_WINDOW",
                IncludeVaults = new[] { "*" },
                RecoveryPointTypes = new[] { "SNAPSHOT" },
                SelectionWindowDays = 8
            }
        });

        _ = new CfnRestoreTestingSelection(this, "RestoreTestSelection", new CfnRestoreTestingSelectionProps
        {
            RestoreTestingPlanName = testPlan.RestoreTestingPlanName,
            RestoreTestingSelectionName = "hl_m12_dynamodb",
            ProtectedResourceType = "DynamoDB",
            ProtectedResourceArns = new[] { table.TableArn },
            IamRoleArn = restoreRole.RoleArn,
            ValidationWindowHours = 1
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m12-backup");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
    }
}

CfnRestoreTestingSelection has a RestoreTestingPlanName property that expects the plan name. If CDK reports a type mismatch in your version, pass the literal string "hl_m12_weekly" and add AddDependency ⚠ VERIFY. The restore testing Lambda is added in the next section.

Complete code

The validation Lambda is the part that turns "restore completed" into "restore proved the data is there". It reads the restored DynamoDB table, compares item count and a sample against expectations, and reports to AWS Backup.

src/Harbourline.Ha.RestoreValidator/Function.cs
using Amazon.Backup;
using Amazon.Backup.Model;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.Core;
using System.Text.Json;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Ha.RestoreValidator;

public sealed class Function(IAmazonBackup backup, IAmazonDynamoDB dynamo)
{
    // Lambda runtime constructor.
    public Function() : this(new AmazonBackupClient(), new AmazonDynamoDBClient()) { }

    public async Task Handler(JsonElement evt, ILambdaContext ctx)
    {
        var detail = evt.GetProperty("detail");
        var status = detail.GetProperty("status").GetString();
        if (status != "COMPLETED") return; // only validate finished restores

        var jobId = detail.GetProperty("restoreJobId").GetString()!;
        var createdArn = detail.GetProperty("createdResourceArn").GetString()!;
        var resourceType = detail.GetProperty("resourceType").GetString();
        if (resourceType != "DynamoDB")
        {
            ctx.Logger.LogInformation($"Skipping {resourceType}: no validator");
            return;
        }

        var tableName = createdArn.Split("table/")[1];
        var (ok, message) = await ValidateAsync(tableName);

        await backup.PutRestoreValidationResultAsync(new PutRestoreValidationResultRequest
        {
            RestoreJobId = jobId,
            ValidationStatus = ok ? RestoreValidationStatus.SUCCESSFUL : RestoreValidationStatus.FAILED,
            ValidationStatusMessage = message
        });
        ctx.Logger.LogInformation($"Restore {jobId}: {(ok ? "SUCCESSFUL" : "FAILED")} - {message}");
    }

    internal async Task<(bool Ok, string Message)> ValidateAsync(string tableName)
    {
        var desc = await dynamo.DescribeTableAsync(tableName);
        if (desc.Table.TableStatus != TableStatus.ACTIVE)
            return (false, $"Table status is {desc.Table.TableStatus}, expected ACTIVE");

        // Item count in DescribeTable is refreshed about every 6 hours, so scan a bounded sample.
        var scan = await dynamo.ScanAsync(new ScanRequest { TableName = tableName, Limit = 25 });
        return RestoreRules.Evaluate(scan.Items.Count, scan.Items.Select(i => i.Keys.ToHashSet()));
    }
}

public static class RestoreRules
{
    private static readonly string[] RequiredAttributes = ["pk", "balance", "currency"];

    public static (bool Ok, string Message) Evaluate(int sampleCount, IEnumerable<HashSet<string>> itemAttributeSets)
    {
        if (sampleCount == 0) return (false, "Restored table is empty");
        var bad = itemAttributeSets.Count(keys => !RequiredAttributes.All(keys.Contains));
        return bad == 0
            ? (true, $"Sampled {sampleCount} items; all have required attributes")
            : (false, $"{bad} of {sampleCount} sampled items miss required attributes");
    }
}

A tiny evidence helper compares each restore job's duration with the service's RTO:

src/Harbourline.Ha.RestoreValidator/RestoreEvidence.cs
namespace Harbourline.Ha.RestoreValidator;

public sealed record RestoreRecord(string JobId, string Resource, TimeSpan Duration, string Validation);

public static class RestoreEvidence
{
    /// <summary>Returns a markdown table line per job, flagging jobs slower than the RTO share.</summary>
    public static IEnumerable<string> Report(IEnumerable<RestoreRecord> jobs, TimeSpan rto, double rtoShare = 0.5)
    {
        // A restore is only one part of recovery: allow it half of the RTO by default.
        var budget = TimeSpan.FromTicks((long)(rto.Ticks * rtoShare));
        foreach (var j in jobs)
        {
            var flag = j.Duration <= budget && j.Validation == "SUCCESSFUL" ? "PASS" : "REVIEW";
            yield return $"| {j.JobId} | {j.Resource} | {j.Duration.TotalMinutes:F1} min | {j.Validation} | {flag} |";
        }
    }
}

An EventBridge rule (CDK C#) connects the event to the function:

infra/cdk/RestoreValidationRule.cs
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.Lambda;
using Constructs;

namespace Harbourline.Ha.Infra;

public static class RestoreValidationRule
{
    public static void Add(Construct scope, IFunction validator)
    {
        _ = new Rule(scope, "RestoreCompleted", new RuleProps
        {
            EventPattern = new EventPattern
            {
                Source = new[] { "aws.backup" },
                DetailType = new[] { "Restore Job State Change" },
                Detail = new Dictionary<string, object>
                {
                    ["status"] = new[] { "COMPLETED" }
                }
            },
            Targets = new IRuleTarget[] { new LambdaFunction(validator) }
        });
    }
}

The Lambda role needs backup:PutRestoreValidationResult, dynamodb:DescribeTable and dynamodb:Scan on arn:aws:dynamodb:*:*:table/awsbackup-restore-test* ⚠ VERIFY the restored table's naming. Put the code in labs/ and unit-test RestoreRules and RestoreEvidence with xUnit v3 without AWS.

Code explained line by line

  1. Function(IAmazonBackup backup, IAmazonDynamoDB dynamo) uses a primary constructor so tests can pass fakes. The parameterless constructor is what the Lambda runtime calls.
  2. The handler ignores events that are not COMPLETED; AWS Backup emits state changes for every transition.
  3. createdResourceArn identifies the restored table. A restore test names the new table itself, so you never guess; split on table/.
  4. ValidateAsync checks that the table is ACTIVE before looking inside. A restored table that is still creating would give a false failure.
  5. The scan is bounded with Limit = 25. A full scan on a large table would be slow and expensive. Item counts from DescribeTable are updated only periodically, so they are poor evidence.
  6. RestoreRules.Evaluate is a pure function that checks required attributes. Real validation would compare a checksum of a known sample (the fingerprint idea from Module 6), a recent known transaction, or the highest booking_date.
  7. PutRestoreValidationResultAsync reports SUCCESSFUL or FAILED, with a short message. If you do not report within the validation window, AWS Backup deletes the restored resource anyway.
  8. RestoreEvidence.Report budgets half of the RTO for the restore step, because failover, DNS and validation also take time.

Security considerations

⚠ Learning shortcut — not for production

The lab uses governance-mode Vault Lock, one account, default keys policy and RemovalPolicy.DESTROY. Production banks use a separate backup account, a Vault Locked vault whose retention comes from the records-retention policy, a KMS key policy that grants only AWS Backup and a break-glass role, and an SCP that denies backup:DeleteBackupVault, backup:DeleteRecoveryPoint and backup:PutBackupVaultLockConfiguration outside the break-glass role.

  • Compliance mode is a one-way door. Use a throwaway account, a minimum 3-day grace time and a very small minimum retention for any experiment, or avoid it in a lab entirely. Do not lock a vault in a shared sandbox.
  • Restore creates real resources with real data. Restore tests should run in an isolated account or at least with tight IAM, no internet path, and the same encryption as production. Restored tables are deleted automatically after the validation window.
  • KMS. Cross-account copy requires the destination vault key to allow the source account. A common failure is a copy job stuck on AccessDenied for KMS.
  • Least privilege on the restore role. Use the AWS-managed restore policy as a starting point, then scope down.
  • Audit. Turn on AWS Backup Audit Manager frameworks for "resources are in a backup plan", "backup plan min frequency", "cross-Region copy" and the restore-time control, and keep reports in an S3 bucket with Object Lock. See S3 protection and Object Lock.
  • UK GDPR and erasure. An immutable backup holds personal data until retention ends. Write down how erasure requests are handled for backed-up data (typically "put beyond use" and expire on schedule). Ask your data-protection officer.

Banking use case

Harbourline's backup policy for important business services:

Service dataSourceLocal retentionCopyLock
Ledger databaseAurora35 days plus continuous backupDR Region, central backup accountCompliance, minimum 2,557 days for monthly recovery points ⚠ legal to confirm
Account indexDynamoDB35 daysDR RegionGovernance
Customer documentsS3Versioning plus BackupAnother accountCompliance, per records policy
Build artefactsECR, S37 daysNoneNone

Every quarter, the operational-resilience team runs the restore testing report: restore time, validation result and comparison with the service's RTO. A failed validation raises an incident ticket automatically through an EventBridge rule. That report is a core exhibit for the regulator conversation. Harbourline does not claim the report equals compliance; it is one piece of evidence the compliance team uses.

Common mistakes

  • Backups in the same account and Region as production. One compromised credential or one Regional event deletes everything. Always copy out.
  • Compliance-mode lock set in a hurry. The grace time is the only undo. Review minimum and maximum retention with legal first and never use "Always" retention in locked vaults.
  • Assuming "backup succeeded" means "restore works". Missing KMS permissions, wrong subnet groups and default VPC assumptions break restores. Restore testing finds these while you are calm.
  • Forgetting the tag in resource selection. A new table without the Backup tag is silently unprotected. Add an AWS Config rule or Audit Manager control for coverage.
  • Retention shorter than detection time. If it takes 40 days to notice quiet corruption, 35-day retention is useless. Keep monthly recovery points longer.
  • Expecting restore testing to touch everything. It supports a list of resource types; confirm yours is included ⚠ VERIFY.

Troubleshooting

SymptomLikely causeFix
Backup job FAILED: access deniedRole cannot be assumed by backup.amazonaws.com or lacks service permissionFix the trust policy and attach the AWS-managed backup policy
Copy job fails with KMS errorDestination key policy does not allow the source, or source key not sharedUpdate key policies on both sides
New backup job rejected by vaultLifecycle is shorter than vault lock minimum or longer than maximumAlign the plan's DeleteAfter with lock min and max
Restore test FAILED: no default VPC or subnetEC2 or RDS restore needs a subnet and the account has no default VPCOverride SubnetId or DBSubnetGroupName in the restore testing selection
Restored resource not deletedawsbackup-restore-test tag removed, or deletion still in progressDelete it manually, check the deletion status with describe-restore-job
Validation never reportedLambda did not receive the event, or the window closedCheck the EventBridge rule pattern and the validation window hours
delete-backup-vault failsVault still contains recovery points or is lockedDelete recovery points (allowed in governance mode with permissions) or wait for retention
Restore testing selection rejectedBoth ARNs and conditions suppliedUse specific ARNs or wildcard plus conditions, not both

Cost and cleanup

Cost: credits. Backup storage is billed per GB-month; a tiny DynamoDB table costs cents. Cross-Region copy adds data transfer. Restore tests carry a per-test charge plus the cost of the restored resource while it exists ⚠ VERIFY on the AWS Backup pricing page. Keep the lab to one tiny table, one backup, one restore test, and delete within the day. Free alternative: design the plan, run the C# validator rules in unit tests, and read the AWS Backup documentation tables; no cloud account needed.

# 1. Remove the Vault Lock first if you used governance mode (compliance mode cannot be removed after grace time).
aws backup delete-backup-vault-lock-configuration --backup-vault-name hl-m12-local \
  --region us-east-1 --profile learn-sandbox

# 2. Delete restore testing selections, then the plan.
aws backup delete-restore-testing-selection --restore-testing-plan-name hl_m12_weekly \
  --restore-testing-selection-name hl_m12_dynamodb --region us-east-1 --profile learn-sandbox
aws backup delete-restore-testing-plan --restore-testing-plan-name hl_m12_weekly \
  --region us-east-1 --profile learn-sandbox

# 3. Delete backup selection and plan.
aws backup list-backup-plans --region us-east-1 --profile learn-sandbox
aws backup list-backup-selections --backup-plan-id PLAN_ID --region us-east-1 --profile learn-sandbox
aws backup delete-backup-selection --backup-plan-id PLAN_ID --selection-id SELECTION_ID \
  --region us-east-1 --profile learn-sandbox
aws backup delete-backup-plan --backup-plan-id PLAN_ID --region us-east-1 --profile learn-sandbox

# 4. Delete every recovery point in both vaults, then the vaults.
aws backup list-recovery-points-by-backup-vault --backup-vault-name hl-m12-local \
  --query "RecoveryPoints[].RecoveryPointArn" --output text --region us-east-1 --profile learn-sandbox
aws backup delete-recovery-point --backup-vault-name hl-m12-local \
  --recovery-point-arn RECOVERY_POINT_ARN --region us-east-1 --profile learn-sandbox
aws backup delete-backup-vault --backup-vault-name hl-m12-local --region us-east-1 --profile learn-sandbox
aws backup delete-backup-vault --backup-vault-name hl-m12-dr --region us-west-2 --profile learn-sandbox

# 5. Delete the DynamoDB table and the KMS key (schedule deletion), then verify.
aws dynamodb delete-table --table-name hl-m12-accounts --region us-east-1 --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --region us-east-1 --profile learn-sandbox --query "ResourceTagMappingList[].ResourceARN"

Repeat step 4's recovery point deletion for hl-m12-dr in us-west-2. If you used CDK or Terraform, cdk destroy or terraform destroy removes the stack, but a vault that still contains recovery points will not delete, so empty it first.

Exercises

  • 🟢 Guided: Create the vault, plan and selection with the CLI for one tagged DynamoDB table, run start-backup-job, and note the time the recovery point appears. Then restore it to a new table with start-restore-job and compare item counts.
  • 🟡 Challenge: Add restore testing for the table and a validation Lambda. Write unit tests for RestoreRules.Evaluate covering an empty table, a table with a missing attribute and a good table.
  • 🔴 Live problem: The compliance team says "we need immutable backups for 7 years, but last quarter's restore test took 3 hours and our RTO is 2 hours". Propose changes (continuous backup, warm standby, restore parallelism, smaller recovery scope) and explain what evidence you would show next quarter.

Assignment

Write a backup and restore standard for one Harbourline service: resources and tags, plan rules, local and copy retention, lock mode and its approval, restore testing frequency, validation checks, and the measured-versus-target RTO table. Include the IaC. Commit it to your portfolio repository.

Solutions

Knowledge check

Interview questions

  • How would you protect backups from a ransomware actor with admin access to production? Cross-account copy to a locked vault, SCPs denying deletion, KMS keys in the backup account, MFA-protected break-glass access, and monitored DeleteRecoveryPoint attempts.
  • How do you prove a backup is usable? Automated restore testing with validation, measured duration compared with the RTO, and a retained report.
  • What are the trade-offs of continuous backup? Lower RPO (point-in-time restore within a retention window) at higher storage cost and not available for every resource ⚠ VERIFY; restores still take time.
  • Explain the risk of compliance-mode Vault Lock. After the grace time it cannot be undone, so mistaken retention values keep costing money and keep data you may have to delete.

Certification practice

Completion checklist

  • I can explain vault, recovery point, plan, selection, Vault Lock and restore testing
  • I know why compliance-mode Vault Lock must be handled carefully
  • I completed the guided exercise and measured one restore time
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • AWS Backup gives one policy for many services: plans, tag selection, lifecycle and copies.
  • Immutable backups need separation: another account, another Region, a locked vault.
  • A backup is a hope until restore testing proves it, with validation and a timing.
  • Compliance-mode Vault Lock is irreversible after the grace time: plan retention with legal first.
  • Keep evidence: restore duration against RTO, validation status, and the audit report.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page