Skip to Content
Cloud Engineering Academy
5. Security for Banking & Finance

Backup security and key rotation: AWS Backup, Vault Lock, cross-account copies and zero-downtime rotation

Make Harbourline's backups survive ransomware and rogue administrators with AWS Backup vault policies, Vault Lock, logically air-gapped vaults and cross-account copies, and rotate KMS keys, secrets, access keys and signing keys without an outage.

advanced⏱ 2 h 30 min🟡 Low costSCS-C03SAA-C03SOA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.Backup 4.1.1.2, AWSSDK.KeyManagementService 4.1.1.1) · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain why backups are a primary target in ransomware and insider attacks, and design a backup architecture that a compromised administrator can't destroy.
  • Use AWS Backup plans, vaults, vault access policies, Vault Lock (governance vs compliance mode, grace time), logically air-gapped vaults, cross-account and cross-Region copies, Organizations backup policies and restore testing.
  • Choose the right KMS strategy for backups, including what happens when a key is disabled or deleted.
  • Rotate KMS keys (automatic, custom period, on-demand, manual via alias), Secrets Manager secrets (single-user vs alternating-users), IAM access keys and asymmetric signing keys without downtime.
  • Audit backup and key posture with a .NET 10 tool that flags unlocked vaults, short retention and keys without rotation.

Prerequisites

  • S3 protection — Object Lock concepts reused by Vault Lock
  • Incident response — backups are your recovery path
  • CloudTrail for banks — the evidence of who changed backups and keys
  • AWS KMS — customer managed keys, key policies and envelope encryption
  • Tools: AWS CLI v2 (learn-sandbox profile), .NET 10 SDK

Why this matters in banking

Modern ransomware crews don't just encrypt servers. In cloud environments they first look for the backups: they delete recovery points, shorten retention, disable the KMS key that protects backups, or copy data out and then delete it. If an attacker (or a malicious insider) holds administrator rights in the account where backups live, a conventional backup is just another resource they can delete.

For Harbourline, losing customer account data or the payments ledger isn't an inconvenience — it's an operational-resilience failure with regulatory, financial and reputational consequences. UK regulators expect firms to stay within impact tolerances for important business services, which requires tested recovery. Record-keeping obligations also mean some data must be retained for years.

Keys have the same double edge. Rotating them limits the damage of a leaked secret and satisfies policy (PCI DSS, for example, requires defined cryptoperiods and key changes for keys it covers). Rotating them badly causes outages: a database password changed while 40 application instances still use the old one, or a signing key replaced before partners trust the new public key.

The idea in plain language

Backups protect against loss (deletion, corruption, ransomware, bad deployments). To protect the backups themselves you need three properties:

  1. Separation — copies live in a different account (and often a different Region) that production administrators can't control.
  2. Immutability — copies can't be deleted or have their retention shortened before they expire, even by an administrator: AWS Backup Vault Lock (and S3 Object Lock for S3-based archives).
  3. Recoverability — you've proven you can restore, with the keys and permissions available, within your recovery time objective (RTO).

Key rotation means replacing cryptographic material or credentials on a schedule or after suspected compromise. "Without downtime" always relies on the same trick: a period when both old and new are valid. AWS KMS does this for you by keeping old key material for decryption; for secrets, access keys and signing keys you design the overlap.

Analogy: Backups are the bank's off-site archive. Separation is putting the archive in a different building with different key-holders. Vault Lock is a time lock that even the archive manager can't override. Restore testing is the fire drill. Key rotation is changing the vault combination: you tell everyone the new code before the old one stops working, and you keep the old combination written down in a sealed envelope for the boxes locked with it.

What problem the AWS services solve

AWS Backup

ConceptWhat it isFacts (checked 2026-11-18)
Backup planSchedules, lifecycle (cold storage, expiry) and copy rulesResources assigned by tags or ARNs
Backup vaultContainer for recovery points, encrypted with a KMS keyEach vault has an access policy
Vault access policyResource policy on the vaultUse Deny on backup:DeleteRecoveryPoint, backup:UpdateRecoveryPointLifecycle, policy changes
Vault Lock — governanceLock removable by principals with sufficient IAM permissionsGood for labs and for "two-person" admin models
Vault Lock — complianceAfter the grace time (minimum 3 days, maximum 36,500), the lock and retention can't be changed or removed by anyone, including AWSA vault with recovery points can't be deleted until they expire; retention set to "always" means forever
Min / max retentionJobs whose retention is outside the range failMin retention can be as low as 1 day; max up to 36,500 days
Logically air-gapped vaultSpecialised vault, always locked in compliance mode, backups stored in an AWS Backup service-owned account; shareable via AWS RAM and Multi-party approvalEncrypted with an AWS owned key by default or a customer managed key; minimum retention at least 7 days
Cross-account / cross-Region copyCopy rules send recovery points to another vaultCross-account copy needs AWS Organizations cross-account backup enabled ⚠ VERIFY current prerequisites
Backup policies (Organizations)Org-wide backup plans pushed to member accountsMembers can't modify them
Restore testingScheduled automated restores with validationProduces evidence for resilience reviews
Backup Audit ManagerFrameworks of controls (for example "resources protected by a vault lock")Part of AWS Backup, separate from the closed AWS Audit Manager service
Legal holdPrevents deletion of specific recovery points regardless of lifecycleFor litigation or investigation

Vault Lock is free; you pay normal backup storage. AWS Backup Vault Lock has been assessed by Cohasset Associates for SEC 17a-4, CFTC and FINRA requirements — US rules; UK applicability is for your compliance team to judge.

Key and secret rotation options

Thing to rotateAWS mechanismDowntime-free because…
KMS symmetric key (AWS_KMS origin)Automatic rotation, period 90–2,560 days (default 365); on-demand rotation up to 25 times per keyKey ID stays the same; KMS keeps all old material and picks the right one to decrypt
KMS key with imported materialOn-demand rotation after importing new materialSame as above
Asymmetric, HMAC, custom key store keysManual rotation: new key + alias switchYou keep the old key for verification/decryption during the overlap
Secrets Manager secretRotation Lambda; single user or alternating users strategy; staging labels AWSCURRENT, AWSPENDING, AWSPREVIOUSAlternating users keeps the previous credential valid during rollout
IAM user access keysTwo keys per user: create new, deploy, deactivate old, deleteTwo keys can be active at once (better: replace with roles)
TLS certificates (ACM public)Managed renewalACM renews automatically when DNS validation stays in place

KMS charges a monthly fee for the first and second rotations of key material held for a key; subsequent rotations aren't billed further. Rotation doesn't re-encrypt existing data or data keys — it changes what new encryption uses.

Honest limitations:

  • Compliance mode is irreversible. Get the retention right; a wrong setting can lock you into years of storage costs.
  • A locked vault doesn't protect the key. If the vault uses a customer managed key and someone disables it or schedules its deletion, your locked recovery points become unrestorable. Protect backup keys as carefully as the vault.
  • Not every service's backups behave the same. Some resource types are fully managed by AWS Backup; others use the service's native snapshots. Check per-service behaviour for copies, encryption and cold storage.
  • Account closure. If an account with a backup vault is closed, AWS deletes vault contents after the 90-day suspension period, even with Vault Lock — another reason for copies in a separate, well-governed account.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Recovery Services / Backup vaultAWS Backup vaultEncrypted with a KMS key you choose; access policy on the vault
Immutable vault (locked)Vault Lock compliance modeGrace time of at least 3 days before it becomes permanent
Soft delete for backups(no direct equivalent) Vault Lock + vault access policy + legal holdPrevention rather than undo
Multi-user authorisation (Resource Guard)Multi-party approval for logically air-gapped vaults; vault policies; SCPsMPA is tied to logically air-gapped vault recovery
Cross-region restore / GRS vaultCross-Region copy rulesExplicit copy actions per plan rule
Azure Policy backup assignmentsOrganizations backup policiesPushed from the management account
Key Vault key auto-rotation policyKMS automatic rotation with a custom periodSame key ID; old versions kept transparently
Key Vault secret rotation (Event Grid + Function)Secrets Manager managed rotation (Lambda)Staging labels manage versions

Architecture

Why three copies? The local vault gives fast restores for everyday mistakes. The central vault in a separate account survives a compromised production admin. The DR-Region copy survives a Regional outage. The logically air-gapped vault adds a copy that is locked by design and can be shared to a clean recovery account via RAM or Multi-party approval, even if the owning account is unusable.

Step-by-step implementation

Classify data and set RPO, RTO and retention

With the business and records management: the ledger needs an RPO of minutes (point-in-time recovery) and retention of years; a reporting cache may need no backup at all. Retention drives the Vault Lock min/max values — get them signed off.

Create dedicated KMS keys for backups

A customer managed key per vault, with automatic rotation on. The key policy allows AWS Backup to use it, lets the backup account's restore role decrypt, and does not let workload administrators schedule deletion. Back that up with an SCP denying kms:ScheduleKeyDeletion and kms:DisableKey on keys tagged Purpose=backup, except for a break-glass role.

Create vaults with access policies

Deny backup:DeleteRecoveryPoint, backup:UpdateRecoveryPointLifecycle, backup:PutBackupVaultAccessPolicy, backup:DeleteBackupVaultAccessPolicy and backup:DeleteBackupVaultLockConfiguration to everyone except a break-glass role. Vault Lock enforces retention; the access policy protects the configuration around it.

Lock the vaults

Start with governance mode while you test. Move central and DR vaults to compliance mode once retention values are agreed. Use the grace time (at least 3 days) to confirm everything works before the lock becomes permanent.

Define backup plans centrally

Use an Organizations backup policy so every production account gets the same plan: frequency, lifecycle and copy rules to the central and DR vaults. Assign resources by tags such as Backup=tier1; use a Config rule to detect untagged data stores.

Test restores automatically

Configure restore testing plans to restore samples regularly into an isolated account and validate them (for example with a Lambda function that runs a checksum query). Keep the results as evidence for operational-resilience reviews.

Monitor and alert

EventBridge rules for DeleteRecoveryPoint, DeleteBackupVaultLockConfiguration, PutBackupVaultAccessPolicy, failed backup and copy jobs, and KMS DisableKey/ScheduleKeyDeletion on backup keys. Send them to the SOC.

Set rotation policies for every key and secret

KMS: automatic rotation on all symmetric customer managed keys with the period your policy requires. Secrets: Secrets Manager rotation with alternating users for databases. IAM users: none for workloads (use roles); if unavoidable, a 90-day rotation with two-key overlap. Signing keys: a documented manual rotation with overlap.

AWS Console instructions

  1. Console → AWS Backup → Backup vaults → Create backup vault. Name hl-m05-vault, encryption key: a new customer managed key (or the default AWS managed key for the free path). Tags Project=cea-lab, Lab=m05-l22.
  2. Open the vault → Access policy → Edit → paste the deny policy from the CLI section.
  3. Backup vaults → Vault locks → Create vault lock. Mode Governance, minimum retention 1 day, maximum 7 days. (In the lab, never choose compliance.)
  4. Dashboard → Create on-demand backup. Resource type DynamoDB, a small test table, vault hl-m05-vault, retention 1 day.
  5. Jobs → Backup jobs — wait for Completed. Then Protected resources → (table) → Restore to a new table name and compare item counts.
  6. Restore testing → Create restore testing plan — read the options; for the lab, a design-only plan is enough.
  7. KMS → Customer managed keys → (key) → Key rotation — enable automatic rotation, set the rotation period, and see the Rotate now (on-demand) option and remaining on-demand rotations.

AWS CLI

PROFILE=learn-sandbox
REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $PROFILE)

# --- KMS key for backups, with a 180-day automatic rotation period ---
KEY_ID=$(aws kms create-key --description "hl-m05 backup key" \
  --tags TagKey=Project,TagValue=cea-lab TagKey=Lab,TagValue=m05-l22 TagKey=Purpose,TagValue=backup \
  --query KeyMetadata.KeyId --output text --profile $PROFILE --region $REGION)
aws kms create-alias --alias-name alias/hl-m05-backup --target-key-id $KEY_ID --profile $PROFILE --region $REGION
aws kms enable-key-rotation --key-id $KEY_ID --rotation-period-in-days 180 --profile $PROFILE --region $REGION
aws kms get-key-rotation-status --key-id $KEY_ID --profile $PROFILE --region $REGION

# --- Vault with access policy and governance-mode lock ---
aws backup create-backup-vault --backup-vault-name hl-m05-vault \
  --encryption-key-arn arn:aws:kms:$REGION:$ACCOUNT:key/$KEY_ID \
  --backup-vault-tags Project=cea-lab,Lab=m05-l22,Environment=learn,Owner=you \
  --profile $PROFILE --region $REGION

cat > vault-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyDestructiveActionsExceptBreakGlass",
    "Effect": "Deny",
    "Principal": "*",
    "Action": [
      "backup:DeleteRecoveryPoint",
      "backup:UpdateRecoveryPointLifecycle",
      "backup:PutBackupVaultAccessPolicy",
      "backup:DeleteBackupVaultAccessPolicy",
      "backup:DeleteBackupVaultLockConfiguration"
    ],
    "Resource": "*",
    "Condition": { "ArnNotLike": { "aws:PrincipalArn": "arn:aws:iam::$ACCOUNT:role/hl-backup-breakglass" } }
  }]
}
EOF
aws backup put-backup-vault-access-policy --backup-vault-name hl-m05-vault \
  --policy file://vault-policy.json --profile $PROFILE --region $REGION

# GOVERNANCE mode = no --changeable-for-days. (Adding it creates a COMPLIANCE lock — never in a lab.)
aws backup put-backup-vault-lock-configuration --backup-vault-name hl-m05-vault \
  --min-retention-days 1 --max-retention-days 7 --profile $PROFILE --region $REGION
aws backup describe-backup-vault --backup-vault-name hl-m05-vault --profile $PROFILE --region $REGION \
  --query "{Locked:Locked,Min:MinRetentionDays,Max:MaxRetentionDays,Key:EncryptionKeyArn}"

# --- On-demand backup of a tiny DynamoDB table, kept for 1 day ---
aws dynamodb create-table --table-name hl-m05-backup-demo --billing-mode PAY_PER_REQUEST \
  --attribute-definitions AttributeName=account_id,AttributeType=S --key-schema AttributeName=account_id,KeyType=HASH \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m05-l22 --profile $PROFILE --region $REGION
aws dynamodb wait table-exists --table-name hl-m05-backup-demo --profile $PROFILE --region $REGION
aws dynamodb put-item --table-name hl-m05-backup-demo --profile $PROFILE --region $REGION \
  --item '{"account_id":{"S":"ACC-0000012345"},"currency":{"S":"GBP"},"balance":{"N":"1520.75"}}'

aws backup start-backup-job --backup-vault-name hl-m05-vault --profile $PROFILE --region $REGION \
  --resource-arn arn:aws:dynamodb:$REGION:$ACCOUNT:table/hl-m05-backup-demo \
  --iam-role-arn arn:aws:iam::$ACCOUNT:role/service-role/AWSBackupDefaultServiceRole \
  --lifecycle DeleteAfterDays=1

aws backup list-backup-jobs --by-backup-vault-name hl-m05-vault --profile $PROFILE --region $REGION \
  --query "BackupJobs[].{State:State,RP:RecoveryPointArn}"

# Try to delete the recovery point: should be DENIED by the vault policy / lock
aws backup delete-recovery-point --backup-vault-name hl-m05-vault \
  --recovery-point-arn <recovery-point-arn> --profile $PROFILE --region $REGION

# --- Rotation commands you will use ---
aws kms rotate-key-on-demand --key-id $KEY_ID --profile $PROFILE --region $REGION     # counts toward the 25 limit
aws kms list-key-rotations --key-id $KEY_ID --profile $PROFILE --region $REGION
aws secretsmanager rotate-secret --secret-id hl/m05/db-app --profile $PROFILE --region $REGION   # if you have one

AWSBackupDefaultServiceRole is created the first time you use the AWS Backup console with default settings; if it doesn't exist, create a role trusted by backup.amazonaws.com with the managed policies AWSBackupServiceRolePolicyForBackup and AWSBackupServiceRolePolicyForRestores.

Logically air-gapped vault (design reference — its minimum retention is at least 7 days, so recovery points cost a week of storage):

aws backup create-logically-air-gapped-backup-vault --backup-vault-name hl-m05-airgapped \
  --min-retention-days 7 --max-retention-days 35 --profile $PROFILE --region $REGION

Infrastructure as Code

The stacks create a backup KMS key with rotation, a vault with an access policy and a governance lock, and a plan that backs up resources tagged Backup=hl-m05 daily with a copy rule (commented) to a central vault.

BackupStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.Backup;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Constructs;

namespace Harbourline.Security.Infra;

public sealed class BackupStack : Stack
{
    public BackupStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        Tags.SetTag("Project", "cea-lab");
        Tags.SetTag("Lab", "m05-l22");
        Tags.SetTag("Environment", "learn");

        var key = new Key(this, "BackupKey", new KeyProps
        {
            Alias = "alias/hl-m05-backup",
            EnableKeyRotation = true,
            RotationPeriod = Duration.Days(180),
            PendingWindow = Duration.Days(7),
            RemovalPolicy = RemovalPolicy.DESTROY,   // learning shortcut; production: RETAIN
        });
        Amazon.CDK.Tags.Of(key).Add("Purpose", "backup");

        var vault = new BackupVault(this, "Vault", new BackupVaultProps
        {
            BackupVaultName = "hl-m05-vault",
            EncryptionKey = key,
            RemovalPolicy = RemovalPolicy.DESTROY,
            // LEARNING SHORTCUT: governance mode (no ChangeableFor). Production central vault:
            // LockConfiguration = new LockConfiguration { MinRetention = Duration.Days(35), MaxRetention = Duration.Days(2557), ChangeableFor = Duration.Days(3) }
            LockConfiguration = new LockConfiguration
            {
                MinRetention = Duration.Days(1),
                MaxRetention = Duration.Days(7),
            },
            AccessPolicy = new PolicyDocument(new PolicyDocumentProps
            {
                Statements =
                [
                    new PolicyStatement(new PolicyStatementProps
                    {
                        Sid = "DenyDestructiveActionsExceptBreakGlass",
                        Effect = Effect.DENY,
                        Principals = [new AnyPrincipal()],
                        Actions =
                        [
                            "backup:DeleteRecoveryPoint",
                            "backup:UpdateRecoveryPointLifecycle",
                            "backup:PutBackupVaultAccessPolicy",
                            "backup:DeleteBackupVaultAccessPolicy",
                            "backup:DeleteBackupVaultLockConfiguration",
                        ],
                        Resources = ["*"],
                        Conditions = new Dictionary<string, object>
                        {
                            ["ArnNotLike"] = new Dictionary<string, string>
                            {
                                ["aws:PrincipalArn"] = $"arn:aws:iam::{Account}:role/hl-backup-breakglass",
                            },
                        },
                    }),
                ],
            }),
        });

        var plan = new BackupPlan(this, "Plan", new BackupPlanProps
        {
            BackupPlanName = "hl-m05-daily",
            BackupPlanRules =
            [
                new BackupPlanRule(new BackupPlanRuleProps
                {
                    RuleName = "daily-0300",
                    BackupVault = vault,
                    ScheduleExpression = Schedule.Cron(new CronOptions { Hour = "3", Minute = "0" }),
                    DeleteAfter = Duration.Days(2),
                    // CopyActions = [new BackupPlanCopyActionProps { DestinationBackupVault = centralVault, DeleteAfter = Duration.Days(35) }],
                }),
            ],
        });

        plan.AddSelection("Tagged", new BackupSelectionOptions
        {
            Resources = [BackupResource.FromTag("Backup", "hl-m05")],
        });
    }
}

Complete code

BackupKeyPosture is a .NET 10 console app that audits one account and Region and prints a posture report:

  • Vaults: locked or not, lock mode inferred from the lock date, min/max retention against policy, KMS key type (AWS managed keys can't be shared cross-account, which blocks some copy designs), and vault type.
  • KMS keys: for each enabled customer managed symmetric key, whether automatic rotation is on and whether the period meets policy; for asymmetric/HMAC keys, a reminder that only manual rotation applies.

The rules are pure functions with tests; the AWS calls are a thin layer.

BackupKeyPosture/BackupKeyPosture.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Security.BackupKeyPosture</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.Backup" Version="4.1.1.2" />
    <PackageReference Include="AWSSDK.KeyManagementService" Version="4.1.1.1" />
  </ItemGroup>
</Project>
BackupKeyPosture/Rules.cs
namespace Harbourline.Security.BackupKeyPosture;

public sealed record VaultInfo(string Name, string VaultType, bool Locked, DateTimeOffset? LockDate,
    long? MinRetentionDays, long? MaxRetentionDays, string? EncryptionKeyArn, bool KeyIsAwsManaged, long RecoveryPoints);

public sealed record KeyInfo(string KeyId, string? Alias, string KeySpec, string Origin, bool RotationEnabled, int? RotationPeriodDays);

public sealed record PostureFinding(string Severity, string Resource, string Message);

public sealed record Policy(int MinRetentionDays = 35, int MaxRotationPeriodDays = 365, bool RequireLock = true);

public static class Rules
{
    public static IEnumerable<PostureFinding> CheckVault(VaultInfo v, Policy p, DateTimeOffset now)
    {
        if (v.VaultType == "LOGICALLY_AIR_GAPPED_BACKUP_VAULT")
        {
            yield return new("INFO", v.Name, "Logically air-gapped vault (always compliance-locked)");
            yield break;
        }

        if (!v.Locked)
        {
            if (p.RequireLock) yield return new("HIGH", v.Name, "Vault is not locked — an administrator can delete recovery points");
        }
        else if (v.LockDate is null)
        {
            yield return new("MEDIUM", v.Name, "Governance-mode lock — removable by privileged IAM principals");
        }
        else if (v.LockDate > now)
        {
            yield return new("INFO", v.Name, $"Compliance lock in grace time until {v.LockDate:u} — still changeable");
        }
        else
        {
            yield return new("INFO", v.Name, $"Compliance lock active since {v.LockDate:u}");
        }

        if (v.Locked && (v.MinRetentionDays ?? 0) < p.MinRetentionDays)
            yield return new("MEDIUM", v.Name, $"Minimum retention {v.MinRetentionDays ?? 0} d is below policy {p.MinRetentionDays} d");

        if (v.Locked && v.MaxRetentionDays is null)
            yield return new("LOW", v.Name, "No maximum retention — mistakes like 'retain forever' cannot be caught");

        if (v.KeyIsAwsManaged)
            yield return new("MEDIUM", v.Name, "Encrypted with an AWS managed key — you can't control its key policy or share it cross-account");
    }

    public static IEnumerable<PostureFinding> CheckKey(KeyInfo k, Policy p)
    {
        var name = k.Alias ?? k.KeyId;
        if (k.KeySpec != "SYMMETRIC_DEFAULT")
        {
            yield return new("INFO", name, $"{k.KeySpec}: automatic rotation not supported — document a manual rotation (new key + alias switch)");
            yield break;
        }
        if (k.Origin == "EXTERNAL")
        {
            yield return new("INFO", name, "Imported key material: rotate on demand after importing new material");
            yield break;
        }
        if (!k.RotationEnabled)
            yield return new("HIGH", name, "Automatic rotation is OFF");
        else if (k.RotationPeriodDays > p.MaxRotationPeriodDays)
            yield return new("MEDIUM", name, $"Rotation period {k.RotationPeriodDays} d exceeds policy {p.MaxRotationPeriodDays} d");
    }
}
BackupKeyPosture/Program.cs
using Amazon.Backup;
using Amazon.Backup.Model;
using Amazon.KeyManagementService;
using Amazon.KeyManagementService.Model;
using Harbourline.Security.BackupKeyPosture;

var policy = new Policy(MinRetentionDays: args.Length > 0 ? int.Parse(args[0]) : 35);
var now = DateTimeOffset.UtcNow;
var findings = new List<PostureFinding>();

using var backup = new AmazonBackupClient();
using var kms = new AmazonKeyManagementServiceClient();

// Map key ARNs/IDs to aliases, and find AWS managed keys (aliases starting with alias/aws/)
var aliases = new List<AliasListEntry>();
string? marker = null;
do
{
    var page = await kms.ListAliasesAsync(new ListAliasesRequest { Marker = marker });
    aliases.AddRange(page.Aliases ?? []);
    marker = page.Truncated == true ? page.NextMarker : null;
} while (marker is not null);
var aliasByKey = aliases.Where(a => a.TargetKeyId is not null)
    .GroupBy(a => a.TargetKeyId!).ToDictionary(g => g.Key, g => g.First().AliasName);

// --- Vaults ---
string? token = null;
do
{
    var page = await backup.ListBackupVaultsAsync(new ListBackupVaultsRequest { NextToken = token });
    foreach (var v in page.BackupVaultList ?? [])
    {
        var keyId = v.EncryptionKeyArn?.Split('/').LastOrDefault();
        var awsManaged = keyId is not null && aliasByKey.TryGetValue(keyId, out var alias) && alias.StartsWith("alias/aws/", StringComparison.Ordinal);
        var info = new VaultInfo(
            v.BackupVaultName, v.VaultType?.Value ?? "BACKUP_VAULT", v.Locked ?? false,
            v.LockDate is { } ld ? new DateTimeOffset(ld, TimeSpan.Zero) : null,
            v.MinRetentionDays, v.MaxRetentionDays, v.EncryptionKeyArn, awsManaged, v.NumberOfRecoveryPoints ?? 0);
        findings.AddRange(Rules.CheckVault(info, policy, now));
    }
    token = page.NextToken;
} while (!string.IsNullOrEmpty(token));

// --- Customer managed keys ---
marker = null;
do
{
    var page = await kms.ListKeysAsync(new ListKeysRequest { Marker = marker });
    foreach (var k in page.Keys ?? [])
    {
        var meta = (await kms.DescribeKeyAsync(new DescribeKeyRequest { KeyId = k.KeyId })).KeyMetadata;
        if (meta.KeyManager != KeyManagerType.CUSTOMER || meta.KeyState != KeyState.Enabled) continue;

        bool enabled = false; int? period = null;
        if (meta.KeySpec == KeySpec.SYMMETRIC_DEFAULT && meta.Origin == OriginType.AWS_KMS)
        {
            var rot = await kms.GetKeyRotationStatusAsync(new GetKeyRotationStatusRequest { KeyId = k.KeyId });
            enabled = rot.KeyRotationEnabled ?? false;
            period = rot.RotationPeriodInDays;
        }
        var info = new KeyInfo(k.KeyId, aliasByKey.GetValueOrDefault(k.KeyId), meta.KeySpec.Value, meta.Origin.Value, enabled, period);
        findings.AddRange(Rules.CheckKey(info, policy));
    }
    marker = page.Truncated == true ? page.NextMarker : null;
} while (marker is not null);

string[] order = ["HIGH", "MEDIUM", "LOW", "INFO"];
foreach (var f in findings.OrderBy(f => Array.IndexOf(order, f.Severity)))
    Console.WriteLine($"{f.Severity,-6} {f.Resource,-40} {f.Message}");

return findings.Any(f => f.Severity == "HIGH") ? 1 : 0;
BackupKeyPosture.Tests/RulesTests.cs
using Harbourline.Security.BackupKeyPosture;
using Xunit;

public sealed class RulesTests
{
    private static readonly DateTimeOffset Now = new(2026, 11, 18, 12, 0, 0, TimeSpan.Zero);
    private static readonly Policy P = new(MinRetentionDays: 35, MaxRotationPeriodDays: 365);

    private static VaultInfo Vault(bool locked, DateTimeOffset? lockDate = null, long? min = 35, long? max = 2557, bool awsKey = false) =>
        new("hl-central", "BACKUP_VAULT", locked, lockDate, min, max, "arn:aws:kms:us-east-1:111122223333:key/abc", awsKey, 10);

    [Fact]
    public void Unlocked_vault_is_high() =>
        Assert.Contains(Rules.CheckVault(Vault(false), P, Now), f => f.Severity == "HIGH");

    [Fact]
    public void Governance_lock_is_medium() =>
        Assert.Contains(Rules.CheckVault(Vault(true, lockDate: null), P, Now), f => f.Message.StartsWith("Governance"));

    [Fact]
    public void Compliance_lock_in_grace_time_is_reported()
    {
        var f = Rules.CheckVault(Vault(true, Now.AddDays(2)), P, Now).ToList();
        Assert.Contains(f, x => x.Message.Contains("grace time"));
        Assert.DoesNotContain(f, x => x.Severity == "HIGH");
    }

    [Fact]
    public void Short_retention_and_aws_managed_key_are_flagged()
    {
        var f = Rules.CheckVault(Vault(true, Now.AddDays(-30), min: 7, awsKey: true), P, Now).ToList();
        Assert.Contains(f, x => x.Message.Contains("below policy"));
        Assert.Contains(f, x => x.Message.Contains("AWS managed key"));
    }

    [Theory]
    [InlineData(false, null, "HIGH")]
    [InlineData(true, 730, "MEDIUM")]
    public void Symmetric_key_rotation(bool enabled, int? period, string severity) =>
        Assert.Contains(Rules.CheckKey(new KeyInfo("k1", "alias/hl-ledger", "SYMMETRIC_DEFAULT", "AWS_KMS", enabled, period), P),
            f => f.Severity == severity);

    [Fact]
    public void Compliant_symmetric_key_has_no_findings() =>
        Assert.Empty(Rules.CheckKey(new KeyInfo("k1", null, "SYMMETRIC_DEFAULT", "AWS_KMS", true, 180), P));

    [Fact]
    public void Asymmetric_key_gets_manual_rotation_reminder() =>
        Assert.Contains(Rules.CheckKey(new KeyInfo("k2", "alias/hl-payment-signing", "ECC_NIST_P256", "AWS_KMS", false, null), P),
            f => f.Message.Contains("manual rotation"));
}

Zero-downtime rotation patterns in .NET

Database credentials with Secrets Manager (alternating users). Two database users (app_a, app_b) take turns. Rotation sets a new password on the user that isn't current, tests it, then moves AWSCURRENT to it. Applications still holding the previous credentials keep working, because that user's password didn't change. Your code should still refresh the secret and retry once on an authentication failure:

RotatingConnectionFactory.cs
using System.Text.Json;
using Amazon.SecretsManager.Extensions.Caching;
using Npgsql;

namespace Harbourline.Payments.Data;

public sealed class RotatingConnectionFactory(SecretsManagerCache cache, string secretId)
{
    public async Task<NpgsqlConnection> OpenAsync(CancellationToken ct)
    {
        try
        {
            return await OpenWithSecretAsync(await cache.GetSecretString(secretId), ct);
        }
        catch (PostgresException ex) when (ex.SqlState == "28P01")   // invalid_password
        {
            // Secret rotated since we cached it: force a refresh and try once more.
            await cache.RefreshNowAsync(secretId);
            return await OpenWithSecretAsync(await cache.GetSecretString(secretId), ct);
        }
    }

    private static async Task<NpgsqlConnection> OpenWithSecretAsync(string secretJson, CancellationToken ct)
    {
        using var doc = JsonDocument.Parse(secretJson);
        var s = doc.RootElement;
        var csb = new NpgsqlConnectionStringBuilder
        {
            Host = s.GetProperty("host").GetString(),
            Port = s.GetProperty("port").GetInt32(),
            Username = s.GetProperty("username").GetString(),
            Password = s.GetProperty("password").GetString(),
            Database = s.GetProperty("dbname").GetString(),
            SslMode = SslMode.VerifyFull,
        };
        var conn = new NpgsqlConnection(csb.ConnectionString);
        await conn.OpenAsync(ct);
        return conn;
    }
}

This uses the AWSSDK.SecretsManager.Caching package (SecretsManagerCache) and Npgsql ⚠ VERIFY RefreshNowAsync and GetSecretString signatures against the caching library version you install.

Asymmetric signing keys (manual rotation with overlap). For a payment-instruction signing key (see the transaction-integrity lesson in Part 6 of the module overview):

  1. Create a new KMS asymmetric key; publish its public key to partners before using it.
  2. Verifiers accept signatures from either key (a key ring keyed by key ID in the signature metadata).
  3. Point the alias alias/hl-payment-signing at the new key; new signatures use it.
  4. After the overlap period (longer than the maximum time a signed instruction can be in flight or need verifying), remove the old key from verifiers; disable it; schedule deletion only when no signature will ever need verifying with it again — often never for audit purposes, so keep it disabled instead.

IAM access keys (if you can't avoid them). Create the second key → deploy it everywhere → confirm via get-access-key-last-used that the old key is no longer used → deactivate the old key → wait → delete. Better: remove the need by using roles, IAM Roles Anywhere or OIDC federation.

Code explained line by line

  1. VaultInfo and KeyInfo are the tool's own models; the AWS SDK types are mapped into them in Program.cs, so the rules don't depend on the SDK and are easy to test.
  2. Policy holds the bank's thresholds (minimum retention, maximum rotation period, lock required) with defaults; a command-line argument overrides the retention for quick experiments.
  3. In CheckVault, logically air-gapped vaults are reported as informational and skipped, because they're always compliance-locked by design.
  4. The lock logic follows the documented API behaviour: an unlocked vault is HIGH; Locked with no LockDate indicates a governance lock; a LockDate in the future means a compliance lock still in grace time (changeable); a past LockDate means immutable ⚠ VERIFY that LockDate is absent for governance locks in your account's API responses.
  5. Short minimum retention and a missing maximum retention are flagged separately: one lets data be deleted too early, the other lets a mistake ("retain forever") lock in cost permanently.
  6. An AWS managed backup key is flagged because you can't edit its key policy — so you can't grant another account permission to use it for cross-account copies or restores.
  7. CheckKey skips non-symmetric keys and imported-material keys with a reminder, because automatic rotation applies only to symmetric keys with AWS_KMS origin.
  8. In Program.cs, aliases are loaded first (paginated with Marker/NextMarker and Truncated), building a key → alias map. AWS managed keys are identified by aliases beginning with alias/aws/.
  9. ListBackupVaultsAsync is paginated with NextToken; v.Locked ?? false and similar guards handle SDK v4's nullable value types.
  10. For keys, DescribeKey filters to enabled customer managed keys (KeyManagerType.CUSTOMER); GetKeyRotationStatus is only called for symmetric AWS_KMS keys, because it isn't meaningful for others.
  11. The report is sorted HIGH → INFO and the exit code is 1 when any HIGH finding exists — usable in a scheduled job or pipeline.
  12. RotatingConnectionFactory catches PostgreSQL error 28P01 (invalid password), forces a cache refresh and retries once — enough to survive a rotation without hammering the database or Secrets Manager.
  13. SslMode = VerifyFull — rotation doesn't excuse weakening transport security; the client still verifies the server certificate and host name.

Security considerations

⚠ Learning shortcut — not for production

The lab keeps the vault in the same account as the data, uses governance mode with 1–7 day retention, and creates a KMS key with a 7-day deletion window and RemovalPolicy.DESTROY. In production: copies in a separate backup account (and DR Region), central vaults in compliance mode with retention agreed with records management and legal, backup keys protected by SCPs from disabling and deletion, an Organizations backup policy, restore testing with stored evidence, and alerts on every destructive or policy-changing backup and KMS API call.

  • Never test compliance mode in a sandbox. Once the grace time ends, you can't delete the vault until every recovery point expires.
  • Protect the key, not just the vault. Disabling or deleting a backup key makes locked backups unrestorable. Deny kms:DisableKey, kms:ScheduleKeyDeletion and key-policy changes on backup keys via SCP and key policy.
  • Separate administrators. The people who administer production must not administer the backup account. Use separate permission sets and require two people for vault or key policy changes.
  • Restore into isolation. After a ransomware incident, restore into a clean account or VPC and scan before reconnecting; backups may contain the attacker's persistence.
  • Watch for "disable AMI" and snapshot sharing. AWS notes that disabling AMIs can block EC2 restores, and that EBS snapshot recovery points can still be shared or tiered via EC2 APIs. Deny ec2:DisableImage, ec2:ModifySnapshotAttribute and ec2:ModifySnapshotTier for everyone except the backup roles that need them.
  • Rotation is not revocation. Rotating a KMS key doesn't make previously exposed data keys or ciphertext safe. For a suspected key compromise, follow the incident-response decision path.
  • Backups are personal data too. UK GDPR applies to backups. Erasure requests and immutable backups conflict; a common approach (subject to legal advice) is to document that erasure takes effect as backups expire and to prevent restored data from re-introducing erased records.

Banking use case

Harbourline's (fictional) backup and key standard for tier-1 systems (payments ledger, accounts, customer master):

ControlSetting
Backup frequencyContinuous backup / PITR for Aurora and DynamoDB; daily snapshots for EBS
CopiesLocal vault (35 days), central vault in the backup account (7 years for ledger, per records policy), DR Region vault (35 days)
ImmutabilityCentral and DR vaults: Vault Lock compliance mode after a 3-day grace time; logically air-gapped vault for the ledger
AccessBackup account administered by the resilience team; production admins have no access; break-glass role with two-person approval
KeysDedicated CMK per vault; automatic rotation every 365 days; SCP prevents disable/delete
Restore testingWeekly automated restore of the ledger to an isolated account with reconciliation checks; quarterly full DR exercise
EvidenceBackup Audit Manager framework reports; restore-test results; CloudTrail for every change
SecretsDatabase credentials in Secrets Manager with alternating-users rotation every 30 days
Signing keysPayment-instruction signing key rotated yearly with a 90-day overlap; old keys disabled, not deleted

Ransomware tabletop (summary): an attacker gains admin in the payments production account, deletes the local vault's recovery points (the attempt is denied by the vault policy; with the policy removed, local copies can be lost), and tries to disable the backup key (denied by SCP). The central copies in the backup account remain, locked in compliance mode; the team restores the ledger into a clean recovery account and reconciles it against the core-banking ledger. The restore takes 3 h 20 min against an impact tolerance of 4 hours — evidence the design works, and a finding that the runbook needs to be faster.

Common mistakes

  • Backups only in the production account → one compromised admin deletes everything → copy to a separate backup account.
  • Compliance-mode lock with "retain forever" → permanent, unstoppable storage cost → set a maximum retention and check plans before the grace time ends.
  • Forgetting the KMS key → locked backups become unrestorable if the key is disabled → protect backup keys with SCPs and key policies.
  • AWS managed key on a vault you want to copy cross-account → copy or restore fails → use a customer managed key with a cross-account key policy.
  • Never testing restores → you discover missing permissions or broken keys during the real incident → automated restore testing.
  • Rotating a database password with a single-user strategy on a busy service → brief authentication failures for every instance holding the old password → alternating users, plus refresh-and-retry in code.
  • Deleting an old signing key right after rotation → old signatures can no longer be verified for audit → disable and keep.

Troubleshooting

SymptomLikely causeFix
Backup or copy job fails with a retention errorLifecycle outside the vault lock's min/max retentionAlign plan lifecycle with the lock range
AccessDenied on cross-account copyDestination vault policy doesn't allow backup:CopyIntoBackupVault, cross-account backup not enabled in Organizations, or KMS key policy missingUpdate the destination vault policy; enable cross-account backup; grant key use
Restore fails with a KMS errorKey disabled, pending deletion, or the restore role can't use itRe-enable/cancel deletion if possible; grant kms:Decrypt and kms:CreateGrant to the role
Can't delete a recovery point in the labVault access policy denies it, or lock min retention not reachedWait for expiry or use the break-glass role; that's the control working
put-backup-vault-lock-configuration created a compliance lock by mistake--changeable-for-days was suppliedDelete the lock during the grace time with delete-backup-vault-lock-configuration
Applications fail briefly after secret rotationSingle-user strategy or no refresh-on-failureSwitch to alternating users; refresh cache and retry once
get-key-rotation-status shows rotation disabled on an asymmetric keyNot supported for asymmetric keysUse manual rotation with an alias

Cost and cleanup

Item (checked 2026-11-18)PriceLab exposure
AWS Backup Vault LockNo additional chargeFree
DynamoDB backup storage via AWS BackupAbout USD 0.10 per GB-month ⚠ VERIFYKilobytes for a day: effectively free
KMS customer managed keyUSD 1 per month, prorated hourly; first and second rotations add a monthly fee eachA few hours ≈ USD 0.01; then pending deletion (no charge while pending ⚠ VERIFY)
On-demand KMS rotationCounts toward the 25-per-key limit and the rotation fee capOptional in the lab
DynamoDB table (on-demand)Pay per requestCents
Logically air-gapped vaultStorage priced per resource type; minimum 7-day retentionDesign only in the lab

Free alternative: use the AWS managed key (aws/backup) instead of creating a CMK, back up the tiny DynamoDB table with 1-day retention, and run the posture tool and its tests.

PROFILE=learn-sandbox
REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $PROFILE)

# 1. Recovery points expire after 1 day (DeleteAfterDays=1). The vault policy denies manual deletion,
#    so either wait, or remove the policy first (allowed in governance mode):
aws backup delete-backup-vault-access-policy --backup-vault-name hl-m05-vault --profile $PROFILE --region $REGION
aws backup delete-backup-vault-lock-configuration --backup-vault-name hl-m05-vault --profile $PROFILE --region $REGION
for rp in $(aws backup list-recovery-points-by-backup-vault --backup-vault-name hl-m05-vault \
            --query "RecoveryPoints[].RecoveryPointArn" --output text --profile $PROFILE --region $REGION); do
  aws backup delete-recovery-point --backup-vault-name hl-m05-vault --recovery-point-arn $rp --profile $PROFILE --region $REGION
done
aws backup delete-backup-vault --backup-vault-name hl-m05-vault --profile $PROFILE --region $REGION

# 2. Table and key
aws dynamodb delete-table --table-name hl-m05-backup-demo --profile $PROFILE --region $REGION
aws kms delete-alias --alias-name alias/hl-m05-backup --profile $PROFILE --region $REGION
aws kms schedule-key-deletion --key-id $KEY_ID --pending-window-in-days 7 --profile $PROFILE --region $REGION

# Or: cdk destroy / terraform destroy (after recovery points are gone)

# 3. Confirm
aws backup list-backup-vaults --query "BackupVaultList[?starts_with(BackupVaultName,'hl-m05')]" --profile $PROFILE --region $REGION
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query "ResourceTagMappingList[].ResourceARN" --profile $PROFILE --region $REGION

The KMS key stays in PendingDeletion for 7 days; you can cancel with cancel-key-deletion if you made a mistake.

Exercises

  • 🟢 Guided: Create the vault with the access policy and a governance lock, back up the DynamoDB table, try to delete the recovery point (observe the denial), restore it to a new table and compare items, then run BackupKeyPosture. Clean up.
  • 🟡 Challenge: Extend BackupKeyPosture to read each vault's access policy (GetBackupVaultAccessPolicy) and report a MEDIUM finding when it doesn't deny backup:DeleteRecoveryPoint. Add tests using sample policy JSON.
  • 🔴 Live problem: The resilience team proposes switching the central vault to compliance mode with min 2,557 days and no max, starting tomorrow. The ledger plan currently has a copy rule with "retain forever" for monthly backups. Review the change and write your recommendation.

Assignment

Write Harbourline's "backup and key-rotation standard" for your portfolio: the three-copy architecture diagram; vault access policy, lock settings and SCPs as code; the key-rotation table for every key and secret type with the overlap strategy for each; a ransomware tabletop script with expected outcomes; and the BackupKeyPosture tool with tests.

Solutions

Knowledge check

Interview questions

  • "How do you protect backups from ransomware in AWS?" — Separate backup account with separate admins; cross-account and cross-Region copies; Vault Lock compliance mode on central vaults; vault access policies denying deletion; SCPs protecting backup keys and backup configuration; logically air-gapped vault for critical data; automated restore testing; alerts on destructive actions.
  • "Does rotating a KMS key re-encrypt my data?" — No. Rotation changes the key material used for new encryption; KMS keeps old material to decrypt existing ciphertext. Data keys and ciphertext aren't re-encrypted, so rotation doesn't remediate an exposed data key.
  • "How would you rotate database credentials for 50 running service instances without downtime?" — Secrets Manager with the alternating-users strategy, client-side caching with refresh-and-retry on authentication failure, monitoring of authentication errors during the rotation window.
  • "What would make you choose compliance mode over governance mode?" — A regulatory or policy requirement that even privileged administrators can't shorten retention, agreed retention values, protection of the key, and acceptance that the setting is irreversible.

Certification practice

Completion checklist

  • I can explain separation, immutability and recoverability for backups
  • I created a vault with an access policy and a governance lock, and saw a delete denied
  • I restored a backup and compared the data
  • I can describe zero-downtime rotation for KMS keys, database secrets, access keys and signing keys
  • I ran BackupKeyPosture and its tests pass
  • I deleted the vault, table and alias, scheduled key deletion, and confirmed nothing tagged remains

Key takeaways

  • Backups are a ransomware target: keep copies in a separate account and Region, lock them with Vault Lock (compliance mode in production), and protect the configuration with vault access policies and SCPs.
  • A locked vault is only as safe as its KMS key — protect backup keys from being disabled or deleted, and test restores automatically.
  • Compliance-mode locks and "retain forever" are irreversible; set min and max retention and use the grace time to test.
  • KMS rotation (automatic 90–2,560 days, or up to 25 on-demand) keeps key IDs and decrypts old data transparently; asymmetric and HMAC keys need manual rotation with an alias and overlap.
  • Zero-downtime rotation always means an overlap: alternating database users, two access keys, two trusted signing keys — plus refresh-and-retry in your .NET code.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page