Playbook: key rotation without downtime
Rotate KMS keys, database secrets, API keys and signing keys without an outage by using overlap periods: KMS automatic rotation versus manual key swap, Secrets Manager staging labels and alternating users, and clients that refresh credentials on failure.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager) · AWS CLI v2 · AWS KMS · AWS Secrets Manager · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Explain the single rule behind every zero-downtime rotation: a period when both old and new are valid.
- Choose between KMS automatic rotation, on-demand rotation and a manual key swap, and say what each does to existing ciphertext.
- Rotate a database secret with Secrets Manager using staging labels and the alternating users strategy, and know why single-user rotation can cause a brief outage.
- Write a .NET client that caches a secret, refreshes it on authentication failure and survives a rotation.
- Plan the rotation of a signing key or a partner API key with an overlap window.
- Use a disable-before-delete canary to learn what still depends on an old key without causing a permanent outage.
- Run an offline simulation that shows failures with and without the overlap and the refresh-on-failure behaviour.
Prerequisites
- Backup security and key rotation and KMS and envelope encryption
- Secrets Manager and Parameter Store and Secrets management in .NET
- Certificates, TLS and mTLS for the certificate side of rotation
- AccessDenied and IAM policy evaluation, for diagnosing rotation-time denials
- Tools: AWS CLI v2 with the
learn-sandboxprofile; .NET 10 SDK for the simulation
Why this matters in banking
Harbourline's security policy says database credentials rotate every 30 days, customer-data encryption keys every year, and the payment partner's API key every 90 days. Auditors ask for evidence that it happens. Engineers dread it, because past rotations have gone wrong: a database password changed while three services still held the old one, and statement generation failed for 40 minutes at month-end; a key was deleted that an archive bucket still needed, and old statements became unreadable.
A rotation is an intentional change to something every dependent system relies on. Done without an overlap, it is a self-inflicted outage. Done with one, it is boring, which is the goal. The audit evidence matters too: who rotated what, when, and proof that nothing broke.
The idea in plain language
Every rotation has four moments:
- Introduce the new credential or key alongside the old one.
- Overlap: both work. Clients and data may be on either.
- Cut over: all clients use the new one. You can prove it (logs, metrics).
- Retire the old one, after a deliberate waiting period.
What differs between cases is who does each step:
| Thing rotated | Introduce | Overlap works because | Retire |
|---|---|---|---|
| KMS symmetric key, automatic rotation | KMS creates new key material | KMS keeps old material to decrypt old ciphertext; same key ID and ARN | Old material is kept by KMS as long as the key exists |
| KMS key, manual swap | You create a new key and point an alias at it | Old key stays enabled for decrypting old data | You disable, then schedule deletion after re-encrypting or verifying no use |
| Database secret, alternating users | Secrets Manager updates the other user's password | Two database users both valid; the secret flips between them | The previous user stays valid until the next rotation |
| Database secret, single user | The password of the one user is changed | No overlap unless clients refresh promptly | None needed, but a gap is possible |
| JWT signing key | Publish the new public key in the key set | Verifiers accept both until old tokens expire | Remove the old key after the longest token lifetime |
| Partner API key | Partner issues a second key | Both keys valid | Revoke the old key after cutover |
Analogy: Changing the combination of a bank vault. If you change it overnight and tell staff in the morning, nobody can open the door until they hear. A safer way is to fit a second lock with the new combination, announce it, let everyone switch, check the log that nobody still uses the old lock, then remove it. Keeping the old lock's key in a sealed envelope for a month is the cautious final step.
What problem the AWS services solve
- KMS automatic rotation replaces the key's backing material on a schedule without changing the key ID, ARN or alias, and keeps old material for decryption. Applications and policies that reference the key do not change. It applies to symmetric encryption keys with KMS-generated key material; asymmetric, HMAC and some imported-material or custom-key-store keys use other approaches (⚠ VERIFY the current support matrix).
- KMS on-demand rotation triggers a rotation immediately (for a suspected exposure or a drill), with a per-key limit on how many you can do (⚠ VERIFY; the module 5 lesson states 25).
- Secrets Manager rotation runs a Lambda function through four steps (create, set, test, finish) and moves staging labels (
AWSPENDING,AWSCURRENT,AWSPREVIOUS) so clients asking forAWSCURRENTget the new secret, andAWSPREVIOUSkeeps the old one for a fallback. - Managed rotation and hosted rotation functions exist for some engines (for example Amazon RDS and the SQL Server, PostgreSQL and MySQL families), so you do not write the Lambda (⚠ VERIFY availability for your engine and version).
- CloudTrail records every
Decrypt,Encrypt,GenerateDataKey,RotateSecretandGetSecretValue, which is how you prove cutover and find laggards.
What AWS does not do for you: it cannot rotate a partner's API key, a signing key you generate yourself, or a password hard-coded in an old configuration file. It also cannot make an application re-read a secret it cached forever.
KMS rotation facts to remember (⚠ VERIFY before quoting)
| Question | Answer |
|---|---|
| Does automatic rotation change the key ID or ARN? | No |
| Is existing ciphertext re-encrypted? | No. KMS keeps old material and decrypts it. New encryptions use the new material |
| Default rotation period for customer managed keys | 365 days, configurable (the module 5 lesson gives 90 to 2,560 days) |
| AWS managed keys | Rotate automatically on an AWS-controlled schedule |
| Does a manual swap re-encrypt data? | No. You must re-encrypt it (ReEncrypt, S3 copy, or a data migration) or keep the old key enabled |
| Deleting a key | Has a mandatory waiting period (7 to 30 days) and is irreversible after it |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Key Vault key rotation policy and new key versions | KMS automatic rotation | In AWS the key ID stays stable and versions are hidden. In Key Vault applications may reference versions |
Key Vault secret versions (/secrets/name/version) | Secrets Manager version stages (AWSCURRENT, AWSPREVIOUS) | Labels rather than version URLs. Clients normally ask for the label |
| Key Vault rotation via Event Grid and a function | Secrets Manager rotation Lambda | Built in: schedule, Lambda and stage-label handling |
| SQL authentication with two logins for zero-downtime rotation | Alternating users strategy | Provided as a ready-made rotation template |
| Managed identities (no secrets to rotate) | IAM roles for tasks and functions | Prefer roles wherever the target supports IAM auth. Fewer secrets, fewer rotations |
IOptionsMonitor reloading config from Key Vault | A secret cache with a TTL and refresh on failure | You write the caching policy, or use a caching library |
| Certificate renewal in App Service managed certificates | ACM automatic renewal | ACM renews public certificates it issued when DNS or load balancer validation still works |
Architecture
Secrets Manager rotation with alternating users
The four moments
Step-by-step implementation
Minutes 0 to 15: inventory and plan
- List everything that depends on the key or credential. For a KMS key: S3 buckets, EBS volumes, RDS instances, Secrets Manager secrets, DynamoDB tables, Lambda environment variables, SQS queues, application code calling the key directly. Use CloudTrail
Decryptevents andkms list-grantsfor the evidence, not memory. - Pick the mechanism from the table above. Prefer automatic KMS rotation and IAM roles over rotating secrets wherever you can.
- Define the overlap window and the success checks. For a secret: how long until every client has refreshed (cache TTL plus a margin)? For a key: what proves no one needs the old one?
- Schedule it outside peak periods but not at a time when nobody can respond, and avoid month-end for the first run. Get change approval and notify teams.
Minutes 15 to 35: KMS key rotation
Automatic: enable rotation on the key (set the period), confirm with get-key-rotation-status, and record the next rotation date. Nothing else changes. To prove the property, encrypt something before, rotate on demand in the sandbox, then decrypt the old ciphertext: it works.
Manual swap (needed for asymmetric keys, HMAC keys, and keys without automatic rotation):
- Create a new key with the same policy and tags.
- Point the alias (
alias/hl-statements) at the new key. Applications that use the alias now encrypt with the new key. Applications that stored the key ID or ARN do not move, so find them. - Keep the old key enabled. Existing ciphertext still needs it.
- Re-encrypt data in the background (S3 batch copy,
ReEncrypt, database column re-encryption) and verify. - Move to the disable canary below.
Minutes 35 to 55: secret rotation
- Create or confirm the rotation Lambda (hosted or managed) with a network path to the database and to Secrets Manager (a VPC endpoint or NAT).
- Choose alternating users for production databases, and make sure the second user exists with the same privileges.
- Turn on rotation with a schedule, and trigger the first rotation manually in a non-production environment, watching client errors.
- Confirm clients: do they cache the secret? For how long? Do they re-fetch on authentication failure? Fix any that do not (see the C# section).
- In production, rotate during a quiet window, monitor connection failures, and check
GetSecretValuecalls in CloudTrail to see which principals refreshed.
Minutes 55 to 70: disable-before-delete canary
For an old KMS key or revoked credential:
- Disable the old key (or revoke the old credential) rather than deleting it. Disabling is reversible within seconds.
- Watch for failures: CloudTrail events with
errorCodesuch asDisabledException, application errors on decrypt, alarms on 5xx. Wait at least one full business cycle that exercises the data (including month-end if the data is used then). - If something breaks, re-enable immediately, find the owner, fix, then retry.
- When clean, schedule deletion with the longest sensible waiting period (7 to 30 days). During that period the key is unusable but can still be cancelled. Only then does the deletion become irreversible.
Minutes 70 to 80: record and automate
Record evidence: dates, key and secret identifiers, the CloudTrail events proving rotation and cutover, the canary period, approvals. Automate the checks (alarm when rotation fails, when a secret has not rotated within its period, when a key without rotation exists), and put the rotation drill in the calendar, because untested rotation is a risk in itself.
AWS Console instructions
- Console → KMS → Customer managed keys → (key) → Key rotation: toggle automatic rotation, set the period, see rotation history and the Rotate key on demand button.
- Console → KMS → (key) → Aliases: for a manual swap, edit the alias to target the new key.
- Console → KMS → (key) → Key actions → Disable key and Schedule key deletion (choose the waiting period 7 to 30 days).
- Console → Secrets Manager → Secrets → (secret) → Rotation: configure the schedule, choose the Lambda, and use Rotate secret immediately. The Overview tab shows version stages.
- Console → CloudTrail → Event history: filter Event source
kms.amazonaws.comand Event nameDecryptwith the key as resource, orsecretsmanager.amazonaws.comandGetSecretValue.
AWS CLI
# 1. Turn on automatic KMS rotation with a chosen period (days) and verify it.
aws kms enable-key-rotation --key-id alias/hl-statements --rotation-period-in-days 365 --profile learn-sandbox
aws kms get-key-rotation-status --key-id alias/hl-statements --profile learn-sandbox
# 2. Rotate immediately (drill or suspected exposure) and list the rotation history.
aws kms rotate-key-on-demand --key-id alias/hl-statements --profile learn-sandbox
aws kms list-key-rotations --key-id alias/hl-statements --profile learn-sandbox
# 3. Manual swap: create the new key, then repoint the alias. Old key stays ENABLED.
NEW=$(aws kms create-key --description "Harbourline statements key, rotation 2026-11" \
--tags TagKey=Project,TagValue=cea-lab --query KeyMetadata.KeyId --output text --profile learn-sandbox)
aws kms update-alias --alias-name alias/hl-statements --target-key-id "$NEW" --profile learn-sandbox
# 4. Who still decrypts with the OLD key? Look up events by resource (replace with the old key ID).
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=<old-key-id> \
--max-results 50 --query 'Events[].[EventTime,EventName,Username]' --output table --profile learn-sandbox
# 5. Canary: DISABLE the old key (reversible). Re-enable at once if something breaks.
aws kms disable-key --key-id <old-key-id> --profile learn-sandbox
# aws kms enable-key --key-id <old-key-id> --profile learn-sandbox
# 6. After a clean canary period: schedule deletion with a waiting period (7 to 30 days).
aws kms schedule-key-deletion --key-id <old-key-id> --pending-window-in-days 30 --profile learn-sandbox
# aws kms cancel-key-deletion --key-id <old-key-id> --profile learn-sandbox
# 7. Secrets Manager: set a 30-day rotation, rotate now, and inspect the version stages (NOT the value).
aws secretsmanager rotate-secret --secret-id hl/statements/db \
--rotation-lambda-arn arn:aws:lambda:us-east-1:111122223333:function:hl-rotate-sqlserver \
--rotation-rules AutomaticallyAfterDays=30 --profile learn-sandbox
aws secretsmanager describe-secret --secret-id hl/statements/db \
--query '[RotationEnabled,LastRotatedDate,VersionIdsToStages]' --profile learn-sandboxAvoid commands that print secret values (get-secret-value with SecretString) in a terminal that is logged or shared. Query version IDs and stages instead.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.SecretsManager;
using Constructs;
public class RotationStack : Stack
{
public RotationStack(Construct scope, string id, IStackProps props, ISecret masterSecret) : base(scope, id, props)
{
// Customer managed key with automatic rotation. Setting the period needs a recent CDK version (⚠ VERIFY the property).
var key = new Key(this, "StatementsKey", new KeyProps
{
Alias = "alias/hl-statements",
Description = "Harbourline statements data key",
EnableKeyRotation = true,
RotationPeriod = Duration.Days(365),
PendingWindow = Duration.Days(30), // waiting period applied if the key is ever deleted
RemovalPolicy = RemovalPolicy.RETAIN
});
// Database secret, rotated every 30 days with the alternating-users (multi-user) hosted rotation.
var dbSecret = new Secret(this, "DbSecret", new SecretProps
{
SecretName = "hl/statements/db",
EncryptionKey = key,
GenerateSecretString = new SecretStringGenerator
{
SecretStringTemplate = "{\"username\":\"hl_app\"}",
GenerateStringKey = "password",
ExcludePunctuation = true
}
});
dbSecret.AddRotationSchedule("Rotation", new RotationScheduleOptions
{
AutomaticallyAfter = Duration.Days(30),
HostedRotation = HostedRotation.SqlServerMultiUser(new MultiUserHostedRotationOptions
{
MasterSecret = masterSecret // secret holding privileged credentials used to alter the second user
})
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-rotation");
}
}Complete code
A simulation of rotation with and without the overlap
This program needs no AWS account. It models a vault, a database that either accepts only the current password (single user) or both the current and previous (alternating users), and a client that caches the secret for an hour. One tick is one minute. A rotation happens at minute 40 and again at minute 80. The program runs four combinations and counts the hard failures.
var scenarios = new[]
{
new Scenario("single user, cache only", AlternatingUsers: false, RetryOnAuthFailure: false),
new Scenario("single user, refresh on failure", AlternatingUsers: false, RetryOnAuthFailure: true),
new Scenario("alternating users, cache only", AlternatingUsers: true, RetryOnAuthFailure: false),
new Scenario("alternating users, refresh on failure", AlternatingUsers: true, RetryOnAuthFailure: true),
};
Console.WriteLine($"{"scenario",-40} {"ok",4} {"hard failures",14} {"refresh retries",16}");
foreach (var s in scenarios)
{
var r = Run(s);
Console.WriteLine($"{s.Name,-40} {r.Ok,4} {r.Failures,14} {r.Retries,16}");
}
return 0;
static Result Run(Scenario s)
{
const int cacheTtlMinutes = 60;
var rotationMinutes = new HashSet<int> { 40, 80 };
// The vault: AWSCURRENT and AWSPREVIOUS passwords.
var generation = 0;
var current = "pw-0";
var previous = "";
// The client cache.
var cached = "";
var fetchedAt = int.MinValue / 2;
int ok = 0, failures = 0, retries = 0;
string Fetch(int minute) { cached = current; fetchedAt = minute; return cached; }
bool DatabaseAccepts(string password) => s.AlternatingUsers ? password == current || password == previous : password == current;
for (var minute = 0; minute < 100; minute++)
{
if (rotationMinutes.Contains(minute))
{
previous = current; // AWSCURRENT becomes AWSPREVIOUS
current = $"pw-{++generation}"; // new AWSCURRENT
}
var password = minute - fetchedAt >= cacheTtlMinutes ? Fetch(minute) : cached;
if (DatabaseAccepts(password)) { ok++; continue; }
if (s.RetryOnAuthFailure)
{
retries++;
password = Fetch(minute); // drop the cache and read AWSCURRENT again
if (DatabaseAccepts(password)) { ok++; continue; }
}
failures++;
}
return new Result(ok, failures, retries);
}
record Scenario(string Name, bool AlternatingUsers, bool RetryOnAuthFailure);
record Result(int Ok, int Failures, int Retries);Create a console project targeting net10.0, save the code as Program.cs and run dotnet run. Expected: "single user, cache only" shows 40 hard failures, "single user, refresh on failure" shows 0 failures with 2 refresh retries, and both alternating-users rows show 0 failures.
A rotation-aware secret provider for a .NET service
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;
public sealed class RotatingSecretProvider(IAmazonSecretsManager client, string secretId, TimeSpan ttl)
{
private readonly SemaphoreSlim _gate = new(1, 1);
private string? _value;
private DateTimeOffset _fetchedAt = DateTimeOffset.MinValue;
// Normal path: return the cached value until it expires (TTL must be MUCH shorter than the rotation interval).
public async Task<string> GetAsync(CancellationToken ct = default)
{
if (_value is not null && DateTimeOffset.UtcNow - _fetchedAt < ttl) return _value;
return await RefreshAsync(ct);
}
// Call this after an authentication failure: drop the cache and read AWSCURRENT again.
public async Task<string> RefreshAsync(CancellationToken ct = default)
{
await _gate.WaitAsync(ct); // avoid a stampede of parallel refreshes
try
{
var response = await client.GetSecretValueAsync(new GetSecretValueRequest
{
SecretId = secretId,
VersionStage = "AWSCURRENT"
}, ct);
_value = response.SecretString;
_fetchedAt = DateTimeOffset.UtcNow;
return _value!;
}
finally { _gate.Release(); }
}
}Use it around the connection logic: build the connection string from GetAsync, and if opening the connection fails with an authentication error (SQL Server error 18456), call RefreshAsync and retry once.
public async Task<SqlConnection> OpenAsync(CancellationToken ct)
{
for (var attempt = 0; ; attempt++)
{
var password = attempt == 0 ? await secrets.GetAsync(ct) : await secrets.RefreshAsync(ct);
var connection = new SqlConnection(BuildConnectionString(password));
try
{
await connection.OpenAsync(ct);
return connection;
}
catch (SqlException ex) when (ex.Number == 18456 && attempt == 0) // login failed: maybe the secret rotated
{
await connection.DisposeAsync(); // refresh and try once more, never loop forever
}
}
}Code explained line by line
- The simulation's
currentandpreviousare the two staging labels. At a rotation the old current becomes previous and a new current is created, the same thingfinishSecretdoes. DatabaseAcceptsis the whole difference between strategies. With a single user, only the newest password works the instant it changes. With alternating users the previous password still belongs to a still-valid database user, so both work.- The client reads the vault only when its cached copy is 60 minutes old. In the single-user, cache-only scenario that means up to 20 minutes of failures after each rotation (minutes 40 to 59 and 80 to 99): 40 in total.
- With
RetryOnAuthFailure, a failed login drops the cache, re-reads, and retries once. Two refreshes (one per rotation) hide the rotation from callers completely. RotatingSecretProvideruses aSemaphoreSlimso a burst of failing requests triggers one refresh, not hundreds of calls to Secrets Manager.VersionStage = "AWSCURRENT"asks for the label rather than a version ID, so the provider follows rotations automatically.- SQL Server login failure is error number 18456. Retrying once after a refresh handles rotation. Looping on a genuine wrong password would lock the account, so the
attempt == 0guard matters. - The cache TTL must be much shorter than the rotation interval: a client that refreshes less often than two rotations apart can hold a credential that is no longer even the previous one.
Security considerations
⚠ Learning shortcut — not for production
The simulation and the sandbox commands rotate disposable credentials and keys. Production rotation touches live payment systems: it needs an approved change, a tested rollback, monitoring, and owners on call. Do not delete a KMS key, or revoke a credential, on the same day you rotate it. Never print secret values into logs, tickets or chat while checking that rotation worked.
- Prefer no secrets. IAM roles, IAM database authentication and managed identities remove the rotation problem. Rotate what you cannot eliminate.
- Rotation Lambda security. It holds privileged database access. Give it a least-privilege role, run it in private subnets, restrict the secret's resource policy, and monitor its errors.
- Key policy and grants during a swap. The new key needs the same key policy, or services and roles lose access. Check grants and cross-account principals.
- Never delete a key to "force" migration. Deleted keys make data permanently unreadable. Disable first, wait a business cycle, then schedule deletion with a long waiting period.
- Log, do not leak. CloudTrail shows who read secrets. Alarm on unexpected
GetSecretValuecallers, and avoid logging secret strings. - Compromise is different from routine rotation. If a credential may be exposed, rotate immediately and also invalidate sessions and investigate. Follow the bank's incident process.
- Compliance. Rotation periods in your policy or standards are set by security and compliance, not by this lesson. PCI DSS key-management requirements apply to cardholder-data keys; confirm with the security team.
Banking use case
For Harbourline, the statements database credential rotates every 30 days with the alternating-users strategy, the statements bucket key rotates automatically every year, and the payment partner API key rotates every 90 days through a documented dual-key window. For the month-end incident that motivated this playbook, the fix was to cache the secret for 15 minutes with a refresh-on-login-failure, switch from a single-user to an alternating-users rotation, and add a drill: each quarter the team runs an on-demand rotation in the pre-production environment during business hours and watches for errors. When the archive bucket's key was swapped manually, the old key was disabled for 45 days, which exposed one forgotten reporting job that still decrypted old archives; the team re-encrypted that data before scheduling deletion.
Common mistakes
- Single-user rotation with long-lived cached passwords. Failures until each client refreshes.
- A cache that never expires, or a TTL longer than the rotation interval.
- Deleting the old key because "we rotated". Old ciphertext needs old material, and a manual swap does not re-encrypt data.
- Referencing key ARNs directly instead of aliases, so a manual swap does not take effect.
- Forgetting cross-account and service principals in the new key's policy.
- Rotation Lambda cannot reach the database or Secrets Manager (network, security group), so rotation fails silently until the alarm.
- Never testing rotation until the audit asks.
- Rotating the partner's key without a dual-key window.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Login failures right after a secret rotation | Clients cached the old password and the strategy had no overlap | Alternating users, shorter TTL, refresh on failure |
| Rotation Lambda times out | No network path to the database or to Secrets Manager | Add VPC endpoint or NAT, check security groups |
AWSPENDING version stuck | A previous rotation failed midway | Check the Lambda logs, fix, run the rotation again (or remove the stale pending label, per documentation) |
AccessDeniedException on kms:Decrypt after a key swap | New key policy missing a principal | Compare policies, add the principals and grants |
| Old objects fail to decrypt | Old key disabled or deleted before data re-encrypted | Re-enable the old key (if not yet deleted), re-encrypt data |
DisabledException appears in CloudTrail after the canary | Something still uses the old key | Re-enable, find the caller, migrate it |
| Rotation did not happen on schedule | Rotation not enabled, or Lambda errors | Check RotationEnabled and LastRotatedDate, add an alarm |
| Token verification errors after signing key change | Verifiers did not have the new public key | Publish the new key in the key set before signing with it |
| Rotation of a partner key breaks calls | No dual-key window | Ask the partner for overlapping keys, schedule cutover |
Cost and cleanup
Cost label: free to low. The simulation is free. In the sandbox, a customer managed KMS key costs a small monthly amount and each rotation of the key adds a small charge (⚠ VERIFY current KMS pricing), and each secret has a monthly charge (⚠ VERIFY current Secrets Manager pricing). Keep any sandbox keys and secrets for hours, not weeks, and schedule deletion afterwards.
# Schedule deletion of sandbox keys (minimum waiting period 7 days), and delete sandbox secrets
aws kms schedule-key-deletion --key-id <sandbox-key-id> --pending-window-in-days 7 --profile learn-sandbox
aws secretsmanager delete-secret --secret-id hl/statements/db --recovery-window-in-days 7 --profile learn-sandbox
aws kms delete-alias --alias-name alias/hl-statements --profile learn-sandbox
# Confirm nothing tagged for this lesson remains active
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-rotation --profile learn-sandboxKMS keys scheduled for deletion and secrets with a recovery window still appear for several days. That is expected.
Exercises
- 🟢 Guided: Run the simulation. Explain each of the four rows. Then change the cache TTL to 10 minutes and note how the single-user, cache-only failures change.
- 🟡 Challenge: Add a fifth scenario: alternating users, no retry, with a TTL of 150 minutes and rotations every 40 minutes. Predict the failures, then run it. What rule does it teach about TTL versus rotation interval?
- 🔴 Live problem: "Security wants the statements bucket key replaced because the old key's policy was too broad. The bucket holds 12 million objects encrypted with SSE-KMS. Some services reference the key by ARN. You cannot have downtime." Write the plan with the steps, checks and rollback.
Assignment
Create a rotation runbook for your portfolio covering one KMS key and one database secret: mechanism chosen and why, overlap window, client behaviour (TTL and refresh on failure), evidence queries from CloudTrail, the disable-before-delete canary and the rollback. Commit as runbooks/key-and-secret-rotation.md with fictional identifiers.
Solutions
Row 1 (single user, cache only): 40 failures. After each rotation the cached password is wrong until the cache expires. Row 2 (single user, refresh on failure): 0 failures and 2 retries, because each first failure triggers a refresh. Row 3 (alternating, cache only): 0 failures because the previous password is still valid for the cached copy. Row 4: 0 failures and 0 retries, nothing ever fails. With a 10-minute TTL the cache refreshes at minutes 10, 20, 30, 40 and so on, so in this exact setup the refresh happens in the same minute as each rotation and the single-user cache-only row shows 0 failures. That is luck of timing: move the rotations to minutes 45 and 85 and you get 5 failures after each. The rule is that the failure window after a rotation is bounded by the TTL, so a shorter TTL shrinks the window but cannot remove it without an overlap or a refresh on failure. Why: only an overlap or refresh-on-failure removes the failure window, shorter caches just narrow it.
With rotations at minutes 40 and 80 and a 150-minute TTL, the client fetches at minute 0 and holds pw-0 for all 100 minutes. After the first rotation (minute 40) pw-0 is previous, so it works. After the second rotation (minute 80) pw-0 is neither current nor previous, so calls from minute 80 to 99 fail: 20 failures even with alternating users. The rule: the cache TTL must be shorter than the time between rotations (at least well under half the interval), because alternating users only protects against one rotation behind. Why: the overlap holds only the immediately previous credential.
Plan: (1) Inventory dependents: list-grants, CloudTrail Decrypt and GenerateDataKey by principal, S3 bucket default encryption settings, other services using the key. (2) Create the new key with a tight, reviewed key policy and the same tags, and add the principals found in step 1. (3) Update the alias and each place that stores the ARN (change those to the alias or the new ARN) in a controlled release. (4) Change the bucket's default encryption to the new key so new objects use it, which has no effect on existing objects. (5) Re-encrypt the 12 million existing objects in the background with an S3 Batch Operations copy job (copy each object onto itself with the new key), at a rate that does not affect users, tracking the completion report. (6) Keep the old key enabled throughout. (7) After completion, verify with an inventory report that no object references the old key, then disable the old key as a canary for a full business cycle including month-end. (8) Schedule deletion with a 30-day window, and keep the evidence. Rollback: point the alias back and re-enable or never disable the old key. Why: a manual swap does not re-encrypt data, so downtime is avoided only by keeping the old key usable until nothing needs it.
Knowledge check
Answer: Have a period when both the old and new credential or key are valid, so no client or data item depends on only the retired one.
Answer: Yes. KMS keeps the old key material and uses it to decrypt older ciphertext. The key ID and ARN stay the same.
Answer: The only valid password changes at the moment of rotation, so any client with the old password cached fails until it refreshes. Alternating users keeps the previous user valid.
Answer: Disabling is reversible and reveals forgotten dependants through errors. Deletion is irreversible after the waiting period and makes data permanently unreadable.
Answer: The overlap covers only the previous credential. A client holding a credential older than that has no valid credential, so it fails.
Interview questions
- How do you rotate a database password with no downtime? Use the alternating-users strategy so two valid users exist, keep clients' cache TTL short, and refresh on login failure.
- What happens to old data when a KMS key rotates automatically? Nothing is re-encrypted. KMS retains the old material for decryption.
- How do you retire an old KMS key safely? Confirm no use through CloudTrail, disable it as a canary for a full business cycle, then schedule deletion with a waiting period.
- How do you rotate a JWT signing key? Publish the new public key in the key set, start signing with it, and remove the old key after the longest token lifetime.
- Why prefer IAM roles to rotated secrets? Short-lived credentials are issued automatically, so there is nothing long-lived to rotate or leak.
Certification practice
A company enables automatic rotation on a symmetric customer managed KMS key. An engineer asks whether existing encrypted S3 objects must be re-encrypted. What is the correct answer?
A. Yes, all objects must be re-encrypted immediately or they become unreadable. B. No. KMS retains the previous key material and can still decrypt older data. C. Yes, but only objects larger than 5 GB. D. No, because rotation deletes the old key.
Answer: B. Automatic rotation keeps old material for decryption and the key ID stays the same. A and C are wrong, and D misstates how rotation works.
An application caches a database password for 24 hours. Secrets Manager rotates the password with a single-user strategy and the application starts failing logins. What is the best fix?
A. Increase the rotation interval. B. Use the alternating-users strategy, shorten the cache TTL and refresh the secret on authentication failure. C. Store the password in an environment variable. D. Disable rotation.
Answer: B. An overlap plus a fresh read after failure removes the outage. A delays but does not fix, C is less secure and does not refresh, and D removes a control.
An operations engineer must retire an old KMS key after a manual key swap, without risking permanent data loss. Which sequence is safest?
A. Delete the key immediately. B. Verify through CloudTrail that nothing uses it, disable it for a full business cycle, then schedule deletion with a waiting period. C. Change the alias only and delete the key at once. D. Rotate the key again and delete it the same day.
Answer: B. Disabling is reversible and surfaces dependants before irreversible deletion. The others risk making data unreadable.
Completion checklist
- I can explain the overlap principle with an example for keys and for secrets
- I can say what KMS automatic rotation does and does not do to old ciphertext
- I ran the simulation and explained each row
- I can describe the disable-before-delete canary
- I know why TTL must be much shorter than the rotation interval
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-rotationremains active
Key takeaways
- Zero-downtime rotation always needs an overlap: introduce, overlap, cut over, retire.
- KMS automatic rotation keeps the key ID and old material, so nothing needs re-encrypting.
- A manual key swap needs the old key kept enabled until data is re-encrypted.
- Alternating users, short caches and refresh-on-failure make secret rotation boring.
- Disable first, wait a business cycle, then delete. Prefer designs with no secrets to rotate.
Further reading
- Rotate AWS KMS keys
- Rotate AWS Secrets Manager secrets
- Previous playbook: SQS messages stuck or redelivered
- Next playbook: Who accessed this customer record?
- Related: KMS AccessDeniedException
- Related: Module 17 overview
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: SQS messages stuck or redelivered
Diagnose SQS messages that never finish, keep reappearing or end up in a dead-letter queue by reading visibility timeout, receive count, consumer errors and FIFO group blocking, then inspect and redrive safely with idempotent handlers.
Playbook: who accessed this customer record?
Answer a data protection team's access question with evidence: scope the request, pull application audit events, CloudTrail data events and database audit logs, join them by AWS request ID, find the access that bypassed the application, and hand over a hashed, reviewable report.