Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: key rotation without downtime

Rotate KMS keys, database secrets, API keys and signing keys without an outage by using overlap periods: KMS automatic rotation versus manual key swap, Secrets Manager staging labels and alternating users, and clients that refresh credentials on failure.

advanced⏱ 1 h 20 min🟢 FreeSCS-C03SOA-C03DVA-C03SAP-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager) · AWS CLI v2 · AWS KMS · AWS Secrets Manager · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Explain the single rule behind every zero-downtime rotation: a period when both old and new are valid.
  • Choose between KMS automatic rotation, on-demand rotation and a manual key swap, and say what each does to existing ciphertext.
  • Rotate a database secret with Secrets Manager using staging labels and the alternating users strategy, and know why single-user rotation can cause a brief outage.
  • Write a .NET client that caches a secret, refreshes it on authentication failure and survives a rotation.
  • Plan the rotation of a signing key or a partner API key with an overlap window.
  • Use a disable-before-delete canary to learn what still depends on an old key without causing a permanent outage.
  • Run an offline simulation that shows failures with and without the overlap and the refresh-on-failure behaviour.

Prerequisites

Why this matters in banking

Harbourline's security policy says database credentials rotate every 30 days, customer-data encryption keys every year, and the payment partner's API key every 90 days. Auditors ask for evidence that it happens. Engineers dread it, because past rotations have gone wrong: a database password changed while three services still held the old one, and statement generation failed for 40 minutes at month-end; a key was deleted that an archive bucket still needed, and old statements became unreadable.

A rotation is an intentional change to something every dependent system relies on. Done without an overlap, it is a self-inflicted outage. Done with one, it is boring, which is the goal. The audit evidence matters too: who rotated what, when, and proof that nothing broke.

The idea in plain language

Every rotation has four moments:

  1. Introduce the new credential or key alongside the old one.
  2. Overlap: both work. Clients and data may be on either.
  3. Cut over: all clients use the new one. You can prove it (logs, metrics).
  4. Retire the old one, after a deliberate waiting period.

What differs between cases is who does each step:

Thing rotatedIntroduceOverlap works becauseRetire
KMS symmetric key, automatic rotationKMS creates new key materialKMS keeps old material to decrypt old ciphertext; same key ID and ARNOld material is kept by KMS as long as the key exists
KMS key, manual swapYou create a new key and point an alias at itOld key stays enabled for decrypting old dataYou disable, then schedule deletion after re-encrypting or verifying no use
Database secret, alternating usersSecrets Manager updates the other user's passwordTwo database users both valid; the secret flips between themThe previous user stays valid until the next rotation
Database secret, single userThe password of the one user is changedNo overlap unless clients refresh promptlyNone needed, but a gap is possible
JWT signing keyPublish the new public key in the key setVerifiers accept both until old tokens expireRemove the old key after the longest token lifetime
Partner API keyPartner issues a second keyBoth keys validRevoke the old key after cutover

Analogy: Changing the combination of a bank vault. If you change it overnight and tell staff in the morning, nobody can open the door until they hear. A safer way is to fit a second lock with the new combination, announce it, let everyone switch, check the log that nobody still uses the old lock, then remove it. Keeping the old lock's key in a sealed envelope for a month is the cautious final step.

What problem the AWS services solve

  • KMS automatic rotation replaces the key's backing material on a schedule without changing the key ID, ARN or alias, and keeps old material for decryption. Applications and policies that reference the key do not change. It applies to symmetric encryption keys with KMS-generated key material; asymmetric, HMAC and some imported-material or custom-key-store keys use other approaches (⚠ VERIFY the current support matrix).
  • KMS on-demand rotation triggers a rotation immediately (for a suspected exposure or a drill), with a per-key limit on how many you can do (⚠ VERIFY; the module 5 lesson states 25).
  • Secrets Manager rotation runs a Lambda function through four steps (create, set, test, finish) and moves staging labels (AWSPENDING, AWSCURRENT, AWSPREVIOUS) so clients asking for AWSCURRENT get the new secret, and AWSPREVIOUS keeps the old one for a fallback.
  • Managed rotation and hosted rotation functions exist for some engines (for example Amazon RDS and the SQL Server, PostgreSQL and MySQL families), so you do not write the Lambda (⚠ VERIFY availability for your engine and version).
  • CloudTrail records every Decrypt, Encrypt, GenerateDataKey, RotateSecret and GetSecretValue, which is how you prove cutover and find laggards.

What AWS does not do for you: it cannot rotate a partner's API key, a signing key you generate yourself, or a password hard-coded in an old configuration file. It also cannot make an application re-read a secret it cached forever.

KMS rotation facts to remember (⚠ VERIFY before quoting)

QuestionAnswer
Does automatic rotation change the key ID or ARN?No
Is existing ciphertext re-encrypted?No. KMS keeps old material and decrypts it. New encryptions use the new material
Default rotation period for customer managed keys365 days, configurable (the module 5 lesson gives 90 to 2,560 days)
AWS managed keysRotate automatically on an AWS-controlled schedule
Does a manual swap re-encrypt data?No. You must re-encrypt it (ReEncrypt, S3 copy, or a data migration) or keep the old key enabled
Deleting a keyHas a mandatory waiting period (7 to 30 days) and is irreversible after it

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Key Vault key rotation policy and new key versionsKMS automatic rotationIn AWS the key ID stays stable and versions are hidden. In Key Vault applications may reference versions
Key Vault secret versions (/secrets/name/version)Secrets Manager version stages (AWSCURRENT, AWSPREVIOUS)Labels rather than version URLs. Clients normally ask for the label
Key Vault rotation via Event Grid and a functionSecrets Manager rotation LambdaBuilt in: schedule, Lambda and stage-label handling
SQL authentication with two logins for zero-downtime rotationAlternating users strategyProvided as a ready-made rotation template
Managed identities (no secrets to rotate)IAM roles for tasks and functionsPrefer roles wherever the target supports IAM auth. Fewer secrets, fewer rotations
IOptionsMonitor reloading config from Key VaultA secret cache with a TTL and refresh on failureYou write the caching policy, or use a caching library
Certificate renewal in App Service managed certificatesACM automatic renewalACM renews public certificates it issued when DNS or load balancer validation still works

Architecture

Secrets Manager rotation with alternating users

The four moments

Step-by-step implementation

Minutes 0 to 15: inventory and plan

  1. List everything that depends on the key or credential. For a KMS key: S3 buckets, EBS volumes, RDS instances, Secrets Manager secrets, DynamoDB tables, Lambda environment variables, SQS queues, application code calling the key directly. Use CloudTrail Decrypt events and kms list-grants for the evidence, not memory.
  2. Pick the mechanism from the table above. Prefer automatic KMS rotation and IAM roles over rotating secrets wherever you can.
  3. Define the overlap window and the success checks. For a secret: how long until every client has refreshed (cache TTL plus a margin)? For a key: what proves no one needs the old one?
  4. Schedule it outside peak periods but not at a time when nobody can respond, and avoid month-end for the first run. Get change approval and notify teams.

Minutes 15 to 35: KMS key rotation

Automatic: enable rotation on the key (set the period), confirm with get-key-rotation-status, and record the next rotation date. Nothing else changes. To prove the property, encrypt something before, rotate on demand in the sandbox, then decrypt the old ciphertext: it works.

Manual swap (needed for asymmetric keys, HMAC keys, and keys without automatic rotation):

  1. Create a new key with the same policy and tags.
  2. Point the alias (alias/hl-statements) at the new key. Applications that use the alias now encrypt with the new key. Applications that stored the key ID or ARN do not move, so find them.
  3. Keep the old key enabled. Existing ciphertext still needs it.
  4. Re-encrypt data in the background (S3 batch copy, ReEncrypt, database column re-encryption) and verify.
  5. Move to the disable canary below.

Minutes 35 to 55: secret rotation

  1. Create or confirm the rotation Lambda (hosted or managed) with a network path to the database and to Secrets Manager (a VPC endpoint or NAT).
  2. Choose alternating users for production databases, and make sure the second user exists with the same privileges.
  3. Turn on rotation with a schedule, and trigger the first rotation manually in a non-production environment, watching client errors.
  4. Confirm clients: do they cache the secret? For how long? Do they re-fetch on authentication failure? Fix any that do not (see the C# section).
  5. In production, rotate during a quiet window, monitor connection failures, and check GetSecretValue calls in CloudTrail to see which principals refreshed.

Minutes 55 to 70: disable-before-delete canary

For an old KMS key or revoked credential:

  1. Disable the old key (or revoke the old credential) rather than deleting it. Disabling is reversible within seconds.
  2. Watch for failures: CloudTrail events with errorCode such as DisabledException, application errors on decrypt, alarms on 5xx. Wait at least one full business cycle that exercises the data (including month-end if the data is used then).
  3. If something breaks, re-enable immediately, find the owner, fix, then retry.
  4. When clean, schedule deletion with the longest sensible waiting period (7 to 30 days). During that period the key is unusable but can still be cancelled. Only then does the deletion become irreversible.

Minutes 70 to 80: record and automate

Record evidence: dates, key and secret identifiers, the CloudTrail events proving rotation and cutover, the canary period, approvals. Automate the checks (alarm when rotation fails, when a secret has not rotated within its period, when a key without rotation exists), and put the rotation drill in the calendar, because untested rotation is a risk in itself.

AWS Console instructions

  1. Console → KMS → Customer managed keys → (key) → Key rotation: toggle automatic rotation, set the period, see rotation history and the Rotate key on demand button.
  2. Console → KMS → (key) → Aliases: for a manual swap, edit the alias to target the new key.
  3. Console → KMS → (key) → Key actions → Disable key and Schedule key deletion (choose the waiting period 7 to 30 days).
  4. Console → Secrets Manager → Secrets → (secret) → Rotation: configure the schedule, choose the Lambda, and use Rotate secret immediately. The Overview tab shows version stages.
  5. Console → CloudTrail → Event history: filter Event source kms.amazonaws.com and Event name Decrypt with the key as resource, or secretsmanager.amazonaws.com and GetSecretValue.

AWS CLI

# 1. Turn on automatic KMS rotation with a chosen period (days) and verify it.
aws kms enable-key-rotation --key-id alias/hl-statements --rotation-period-in-days 365 --profile learn-sandbox
aws kms get-key-rotation-status --key-id alias/hl-statements --profile learn-sandbox

# 2. Rotate immediately (drill or suspected exposure) and list the rotation history.
aws kms rotate-key-on-demand --key-id alias/hl-statements --profile learn-sandbox
aws kms list-key-rotations --key-id alias/hl-statements --profile learn-sandbox

# 3. Manual swap: create the new key, then repoint the alias. Old key stays ENABLED.
NEW=$(aws kms create-key --description "Harbourline statements key, rotation 2026-11" \
  --tags TagKey=Project,TagValue=cea-lab --query KeyMetadata.KeyId --output text --profile learn-sandbox)
aws kms update-alias --alias-name alias/hl-statements --target-key-id "$NEW" --profile learn-sandbox

# 4. Who still decrypts with the OLD key? Look up events by resource (replace with the old key ID).
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=<old-key-id> \
  --max-results 50 --query 'Events[].[EventTime,EventName,Username]' --output table --profile learn-sandbox

# 5. Canary: DISABLE the old key (reversible). Re-enable at once if something breaks.
aws kms disable-key --key-id <old-key-id> --profile learn-sandbox
# aws kms enable-key --key-id <old-key-id> --profile learn-sandbox

# 6. After a clean canary period: schedule deletion with a waiting period (7 to 30 days).
aws kms schedule-key-deletion --key-id <old-key-id> --pending-window-in-days 30 --profile learn-sandbox
# aws kms cancel-key-deletion --key-id <old-key-id> --profile learn-sandbox

# 7. Secrets Manager: set a 30-day rotation, rotate now, and inspect the version stages (NOT the value).
aws secretsmanager rotate-secret --secret-id hl/statements/db \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:111122223333:function:hl-rotate-sqlserver \
  --rotation-rules AutomaticallyAfterDays=30 --profile learn-sandbox
aws secretsmanager describe-secret --secret-id hl/statements/db \
  --query '[RotationEnabled,LastRotatedDate,VersionIdsToStages]' --profile learn-sandbox

Avoid commands that print secret values (get-secret-value with SecretString) in a terminal that is logged or shared. Query version IDs and stages instead.

Infrastructure as Code

using Amazon.CDK;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.SecretsManager;
using Constructs;

public class RotationStack : Stack
{
    public RotationStack(Construct scope, string id, IStackProps props, ISecret masterSecret) : base(scope, id, props)
    {
        // Customer managed key with automatic rotation. Setting the period needs a recent CDK version (⚠ VERIFY the property).
        var key = new Key(this, "StatementsKey", new KeyProps
        {
            Alias = "alias/hl-statements",
            Description = "Harbourline statements data key",
            EnableKeyRotation = true,
            RotationPeriod = Duration.Days(365),
            PendingWindow = Duration.Days(30),        // waiting period applied if the key is ever deleted
            RemovalPolicy = RemovalPolicy.RETAIN
        });

        // Database secret, rotated every 30 days with the alternating-users (multi-user) hosted rotation.
        var dbSecret = new Secret(this, "DbSecret", new SecretProps
        {
            SecretName = "hl/statements/db",
            EncryptionKey = key,
            GenerateSecretString = new SecretStringGenerator
            {
                SecretStringTemplate = "{\"username\":\"hl_app\"}",
                GenerateStringKey = "password",
                ExcludePunctuation = true
            }
        });

        dbSecret.AddRotationSchedule("Rotation", new RotationScheduleOptions
        {
            AutomaticallyAfter = Duration.Days(30),
            HostedRotation = HostedRotation.SqlServerMultiUser(new MultiUserHostedRotationOptions
            {
                MasterSecret = masterSecret            // secret holding privileged credentials used to alter the second user
            })
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-rotation");
    }
}

Complete code

A simulation of rotation with and without the overlap

This program needs no AWS account. It models a vault, a database that either accepts only the current password (single user) or both the current and previous (alternating users), and a client that caches the secret for an hour. One tick is one minute. A rotation happens at minute 40 and again at minute 80. The program runs four combinations and counts the hard failures.

RotationSimulation/Program.cs
var scenarios = new[]
{
    new Scenario("single user, cache only",              AlternatingUsers: false, RetryOnAuthFailure: false),
    new Scenario("single user, refresh on failure",      AlternatingUsers: false, RetryOnAuthFailure: true),
    new Scenario("alternating users, cache only",        AlternatingUsers: true,  RetryOnAuthFailure: false),
    new Scenario("alternating users, refresh on failure", AlternatingUsers: true,  RetryOnAuthFailure: true),
};

Console.WriteLine($"{"scenario",-40} {"ok",4} {"hard failures",14} {"refresh retries",16}");
foreach (var s in scenarios)
{
    var r = Run(s);
    Console.WriteLine($"{s.Name,-40} {r.Ok,4} {r.Failures,14} {r.Retries,16}");
}
return 0;

static Result Run(Scenario s)
{
    const int cacheTtlMinutes = 60;
    var rotationMinutes = new HashSet<int> { 40, 80 };

    // The vault: AWSCURRENT and AWSPREVIOUS passwords.
    var generation = 0;
    var current = "pw-0";
    var previous = "";

    // The client cache.
    var cached = "";
    var fetchedAt = int.MinValue / 2;

    int ok = 0, failures = 0, retries = 0;

    string Fetch(int minute) { cached = current; fetchedAt = minute; return cached; }
    bool DatabaseAccepts(string password) => s.AlternatingUsers ? password == current || password == previous : password == current;

    for (var minute = 0; minute < 100; minute++)
    {
        if (rotationMinutes.Contains(minute))
        {
            previous = current;                // AWSCURRENT becomes AWSPREVIOUS
            current = $"pw-{++generation}";    // new AWSCURRENT
        }

        var password = minute - fetchedAt >= cacheTtlMinutes ? Fetch(minute) : cached;

        if (DatabaseAccepts(password)) { ok++; continue; }

        if (s.RetryOnAuthFailure)
        {
            retries++;
            password = Fetch(minute);          // drop the cache and read AWSCURRENT again
            if (DatabaseAccepts(password)) { ok++; continue; }
        }
        failures++;
    }
    return new Result(ok, failures, retries);
}

record Scenario(string Name, bool AlternatingUsers, bool RetryOnAuthFailure);
record Result(int Ok, int Failures, int Retries);

Create a console project targeting net10.0, save the code as Program.cs and run dotnet run. Expected: "single user, cache only" shows 40 hard failures, "single user, refresh on failure" shows 0 failures with 2 refresh retries, and both alternating-users rows show 0 failures.

A rotation-aware secret provider for a .NET service

RotatingSecretProvider.cs
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;

public sealed class RotatingSecretProvider(IAmazonSecretsManager client, string secretId, TimeSpan ttl)
{
    private readonly SemaphoreSlim _gate = new(1, 1);
    private string? _value;
    private DateTimeOffset _fetchedAt = DateTimeOffset.MinValue;

    // Normal path: return the cached value until it expires (TTL must be MUCH shorter than the rotation interval).
    public async Task<string> GetAsync(CancellationToken ct = default)
    {
        if (_value is not null && DateTimeOffset.UtcNow - _fetchedAt < ttl) return _value;
        return await RefreshAsync(ct);
    }

    // Call this after an authentication failure: drop the cache and read AWSCURRENT again.
    public async Task<string> RefreshAsync(CancellationToken ct = default)
    {
        await _gate.WaitAsync(ct);                      // avoid a stampede of parallel refreshes
        try
        {
            var response = await client.GetSecretValueAsync(new GetSecretValueRequest
            {
                SecretId = secretId,
                VersionStage = "AWSCURRENT"
            }, ct);
            _value = response.SecretString;
            _fetchedAt = DateTimeOffset.UtcNow;
            return _value!;
        }
        finally { _gate.Release(); }
    }
}

Use it around the connection logic: build the connection string from GetAsync, and if opening the connection fails with an authentication error (SQL Server error 18456), call RefreshAsync and retry once.

DbConnectionFactory.cs (excerpt)
public async Task<SqlConnection> OpenAsync(CancellationToken ct)
{
    for (var attempt = 0; ; attempt++)
    {
        var password = attempt == 0 ? await secrets.GetAsync(ct) : await secrets.RefreshAsync(ct);
        var connection = new SqlConnection(BuildConnectionString(password));
        try
        {
            await connection.OpenAsync(ct);
            return connection;
        }
        catch (SqlException ex) when (ex.Number == 18456 && attempt == 0)   // login failed: maybe the secret rotated
        {
            await connection.DisposeAsync();            // refresh and try once more, never loop forever
        }
    }
}

Code explained line by line

  1. The simulation's current and previous are the two staging labels. At a rotation the old current becomes previous and a new current is created, the same thing finishSecret does.
  2. DatabaseAccepts is the whole difference between strategies. With a single user, only the newest password works the instant it changes. With alternating users the previous password still belongs to a still-valid database user, so both work.
  3. The client reads the vault only when its cached copy is 60 minutes old. In the single-user, cache-only scenario that means up to 20 minutes of failures after each rotation (minutes 40 to 59 and 80 to 99): 40 in total.
  4. With RetryOnAuthFailure, a failed login drops the cache, re-reads, and retries once. Two refreshes (one per rotation) hide the rotation from callers completely.
  5. RotatingSecretProvider uses a SemaphoreSlim so a burst of failing requests triggers one refresh, not hundreds of calls to Secrets Manager.
  6. VersionStage = "AWSCURRENT" asks for the label rather than a version ID, so the provider follows rotations automatically.
  7. SQL Server login failure is error number 18456. Retrying once after a refresh handles rotation. Looping on a genuine wrong password would lock the account, so the attempt == 0 guard matters.
  8. The cache TTL must be much shorter than the rotation interval: a client that refreshes less often than two rotations apart can hold a credential that is no longer even the previous one.

Security considerations

⚠ Learning shortcut — not for production

The simulation and the sandbox commands rotate disposable credentials and keys. Production rotation touches live payment systems: it needs an approved change, a tested rollback, monitoring, and owners on call. Do not delete a KMS key, or revoke a credential, on the same day you rotate it. Never print secret values into logs, tickets or chat while checking that rotation worked.

  • Prefer no secrets. IAM roles, IAM database authentication and managed identities remove the rotation problem. Rotate what you cannot eliminate.
  • Rotation Lambda security. It holds privileged database access. Give it a least-privilege role, run it in private subnets, restrict the secret's resource policy, and monitor its errors.
  • Key policy and grants during a swap. The new key needs the same key policy, or services and roles lose access. Check grants and cross-account principals.
  • Never delete a key to "force" migration. Deleted keys make data permanently unreadable. Disable first, wait a business cycle, then schedule deletion with a long waiting period.
  • Log, do not leak. CloudTrail shows who read secrets. Alarm on unexpected GetSecretValue callers, and avoid logging secret strings.
  • Compromise is different from routine rotation. If a credential may be exposed, rotate immediately and also invalidate sessions and investigate. Follow the bank's incident process.
  • Compliance. Rotation periods in your policy or standards are set by security and compliance, not by this lesson. PCI DSS key-management requirements apply to cardholder-data keys; confirm with the security team.

Banking use case

For Harbourline, the statements database credential rotates every 30 days with the alternating-users strategy, the statements bucket key rotates automatically every year, and the payment partner API key rotates every 90 days through a documented dual-key window. For the month-end incident that motivated this playbook, the fix was to cache the secret for 15 minutes with a refresh-on-login-failure, switch from a single-user to an alternating-users rotation, and add a drill: each quarter the team runs an on-demand rotation in the pre-production environment during business hours and watches for errors. When the archive bucket's key was swapped manually, the old key was disabled for 45 days, which exposed one forgotten reporting job that still decrypted old archives; the team re-encrypted that data before scheduling deletion.

Common mistakes

  • Single-user rotation with long-lived cached passwords. Failures until each client refreshes.
  • A cache that never expires, or a TTL longer than the rotation interval.
  • Deleting the old key because "we rotated". Old ciphertext needs old material, and a manual swap does not re-encrypt data.
  • Referencing key ARNs directly instead of aliases, so a manual swap does not take effect.
  • Forgetting cross-account and service principals in the new key's policy.
  • Rotation Lambda cannot reach the database or Secrets Manager (network, security group), so rotation fails silently until the alarm.
  • Never testing rotation until the audit asks.
  • Rotating the partner's key without a dual-key window.

Troubleshooting

SymptomLikely causeFix
Login failures right after a secret rotationClients cached the old password and the strategy had no overlapAlternating users, shorter TTL, refresh on failure
Rotation Lambda times outNo network path to the database or to Secrets ManagerAdd VPC endpoint or NAT, check security groups
AWSPENDING version stuckA previous rotation failed midwayCheck the Lambda logs, fix, run the rotation again (or remove the stale pending label, per documentation)
AccessDeniedException on kms:Decrypt after a key swapNew key policy missing a principalCompare policies, add the principals and grants
Old objects fail to decryptOld key disabled or deleted before data re-encryptedRe-enable the old key (if not yet deleted), re-encrypt data
DisabledException appears in CloudTrail after the canarySomething still uses the old keyRe-enable, find the caller, migrate it
Rotation did not happen on scheduleRotation not enabled, or Lambda errorsCheck RotationEnabled and LastRotatedDate, add an alarm
Token verification errors after signing key changeVerifiers did not have the new public keyPublish the new key in the key set before signing with it
Rotation of a partner key breaks callsNo dual-key windowAsk the partner for overlapping keys, schedule cutover

Cost and cleanup

Cost label: free to low. The simulation is free. In the sandbox, a customer managed KMS key costs a small monthly amount and each rotation of the key adds a small charge (⚠ VERIFY current KMS pricing), and each secret has a monthly charge (⚠ VERIFY current Secrets Manager pricing). Keep any sandbox keys and secrets for hours, not weeks, and schedule deletion afterwards.

# Schedule deletion of sandbox keys (minimum waiting period 7 days), and delete sandbox secrets
aws kms schedule-key-deletion --key-id <sandbox-key-id> --pending-window-in-days 7 --profile learn-sandbox
aws secretsmanager delete-secret --secret-id hl/statements/db --recovery-window-in-days 7 --profile learn-sandbox
aws kms delete-alias --alias-name alias/hl-statements --profile learn-sandbox

# Confirm nothing tagged for this lesson remains active
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-rotation --profile learn-sandbox

KMS keys scheduled for deletion and secrets with a recovery window still appear for several days. That is expected.

Exercises

  • 🟢 Guided: Run the simulation. Explain each of the four rows. Then change the cache TTL to 10 minutes and note how the single-user, cache-only failures change.
  • 🟡 Challenge: Add a fifth scenario: alternating users, no retry, with a TTL of 150 minutes and rotations every 40 minutes. Predict the failures, then run it. What rule does it teach about TTL versus rotation interval?
  • 🔴 Live problem: "Security wants the statements bucket key replaced because the old key's policy was too broad. The bucket holds 12 million objects encrypted with SSE-KMS. Some services reference the key by ARN. You cannot have downtime." Write the plan with the steps, checks and rollback.

Assignment

Create a rotation runbook for your portfolio covering one KMS key and one database secret: mechanism chosen and why, overlap window, client behaviour (TTL and refresh on failure), evidence queries from CloudTrail, the disable-before-delete canary and the rollback. Commit as runbooks/key-and-secret-rotation.md with fictional identifiers.

Solutions

Knowledge check

Interview questions

  • How do you rotate a database password with no downtime? Use the alternating-users strategy so two valid users exist, keep clients' cache TTL short, and refresh on login failure.
  • What happens to old data when a KMS key rotates automatically? Nothing is re-encrypted. KMS retains the old material for decryption.
  • How do you retire an old KMS key safely? Confirm no use through CloudTrail, disable it as a canary for a full business cycle, then schedule deletion with a waiting period.
  • How do you rotate a JWT signing key? Publish the new public key in the key set, start signing with it, and remove the old key after the longest token lifetime.
  • Why prefer IAM roles to rotated secrets? Short-lived credentials are issued automatically, so there is nothing long-lived to rotate or leak.

Certification practice

Completion checklist

  • I can explain the overlap principle with an example for keys and for secrets
  • I can say what KMS automatic rotation does and does not do to old ciphertext
  • I ran the simulation and explained each row
  • I can describe the disable-before-delete canary
  • I know why TTL must be much shorter than the rotation interval
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-rotation remains active

Key takeaways

  • Zero-downtime rotation always needs an overlap: introduce, overlap, cut over, retire.
  • KMS automatic rotation keeps the key ID and old material, so nothing needs re-encrypting.
  • A manual key swap needs the old key kept enabled until data is re-encrypted.
  • Alternating users, short caches and refresh-on-failure make secret rotation boring.
  • Disable first, wait a business cycle, then delete. Prefer designs with no secrets to rotate.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page