Skip to Content
Cloud Engineering Academy
3. .NET Application Development on AWS

Lab 3.5: Parameter Store and Secrets Manager with caching and rotation

Layer configuration from appsettings, environment variables and Parameter Store, change a feature flag without a restart, read a pseudonymisation key from Secrets Manager through the client-side cache, rotate it, and see exactly when the API notices.

intermediate⏱ 1 h 15 min🟡 Low costDVA-C03SCS-C03

Assumes: .NET 10 LTS · Amazon.Extensions.Configuration.SystemsManager 7.1.1 · AWSSDK.SecretsManager 4.1.1.2 · AWSSDK.SecretsManager.Caching 3.0.0 · Amazon.CDK.Lib 2.277.0 · Terraform 1.14 with AWS provider 6.x · AWS CLI v2 · Region us-east-1 · labs/m03-accounts-api. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lab you can:

  • Explain the order in which the Accounts API layers configuration, and show which layer won for a given setting.
  • Store a plain setting as a String parameter and a key as a SecureString, and load both into IConfiguration with ParameterStore:Path.
  • Change a feature flag in Parameter Store and watch a running API pick it up without a restart, using IOptionsMonitor.
  • Read a secret from Secrets Manager through the client-side cache, and prove (with a test) that many requests cause one billable call.
  • Rotate the secret and describe exactly when the running API sees the new value, and what that does to a pseudonymisation key.
  • Write the least-privilege IAM statement a deployed host needs, and tear everything down.

Prerequisites

  • Configuration done right and Secrets — the concepts this lab exercises
  • AWS clients and dependency injection — AddAWSService and singleton clients
  • Docker or the .NET 10 SDK; AWS CLI v2 signed in: aws sso login --profile learn-sandbox
  • Nothing from Labs 3.3 or 3.4 is needed. This lab runs the API on your machine, using your SSO credentials, so there is no host to deploy.

Why this matters in banking

Two audit questions come up in every review: "where do your applications keep their secrets?" and "how do you change a setting in production, and who can see the change?" A bank needs answers that are boring and repeatable: secrets live in a managed store, encrypted, readable only by the role that needs them, rotated on a schedule; plain settings change through a reviewed process and take effect predictably. This lab builds the application side of both answers, and its last third shows the subtle part nobody warns you about: rotating a key that other systems depend on.

The idea in plain language

An ASP.NET Core app builds its configuration from a list of sources; later sources override earlier ones. The Accounts API's list is:

  1. appsettings.json
  2. appsettings.{Environment}.json
  3. environment variables
  4. Parameter Store (only if ParameterStore:Path is set)
  5. AppConfig (only if its three ids are set)

Parameter Store is a hierarchical key-value store: a parameter named /hl-m03/accounts-api/Features/IncludePendingTransactions becomes the configuration key Features:IncludePendingTransactions (slashes become colons after the path prefix is removed). A SecureString parameter is encrypted with KMS; the provider decrypts it for you if your role is allowed.

Secrets Manager stores secrets with versions and staging labels (AWSCURRENT, AWSPREVIOUS) and supports rotation. There is no first-party IConfiguration provider for it, so the lab reads it on demand through AWSSDK.SecretsManager.Caching, which keeps the value in memory for a time-to-live (TTL) so the API does not call AWS on every request.

Analogy: appsettings.json is the printed branch handbook. Environment variables are the sticky notes the branch manager adds for this week. Parameter Store is the intranet page head office edits: you re-read it every few minutes. Secrets Manager is the safe: you do not open it for every customer, you take out what you need, keep it in your pocket for a while (the cache), and go back to the safe when the pocket copy is too old.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalent in this labWhat's different
appsettings.json + environment + Azure App Configuration providerAddSystemsManager(path) on builder.ConfigurationSame layering idea; the provider polls on a timer (reloadAfter) rather than using a push refresh sentinel
Key Vault configuration provider (AddAzureKeyVault)SecretsManagerCache called from a serviceNo IConfiguration provider: you decide the TTL and the failure behaviour ⚠ VERIFY nothing first-party exists in your package set
Feature management (IFeatureManager)IOptionsMonitor<FeatureOptions> over a Parameter Store parameter (simple), or AWS AppConfig feature flags (gradual rollout)The lab keeps the simple one; the module covers AppConfig
Managed identity authorised on the vaultIAM role with ssm:GetParametersByPath / secretsmanager:GetSecretValue on one ARNAuthorisation is an IAM policy on the role, not a vault access policy
Key Vault secret versions and rotation policySecrets Manager versions, staging labels and rotation LambdaRotation is code you (or a template) deploy

Architecture

Security boundary view. The values never appear in the app's configuration files or its container image. The only thing the process holds is temporary credentials from your SSO profile (and, once deployed, from the host's role). SecureString values and secret values are decrypted in memory and never logged: the API logs a cref_... reference computed from the key, not the key.

What is already in the lab folder

FileRole in this lab
src/.../Hosting/HostingExtensions.csAddAwsConfigurationSources: Parameter Store with ParameterStore:ReloadAfter (default 5 minutes, minimum 5 seconds), AppConfig when configured
src/.../Security/Pseudonymisation.csPseudonymisationOptions, ConfigurationKeySource, SecretsManagerKeySource (cache) and the HMAC-SHA256 CustomerPseudonymiser
src/.../Endpoints/AccountEndpoints.csReads IOptionsMonitor<FeatureOptions> per request; logs CustomerRef for GET /v1/customers/{id}/accounts
tests/.../ConfigurationAndLambdaTests.csTests for the wiring below, using fakes only
infra/terraform/config.tfThe parameter, the secret container and the IAM statement, as Terraform

Step-by-step

All commands run from labs/m03-accounts-api in Git Bash.

Run the wiring tests

dotnet test --solution AccountsApi.slnx

Expected: 71 passed, 1 skipped. The tests that belong to this lab:

TestWhat it proves
Parameter_reload_interval_is_parsed_with_a_safe_floorParameterStore:ReloadAfter accepts 00:00:20; blanks, nonsense and anything under 5 seconds fall back to 5 minutes
Without_settings_no_aws_configuration_source_is_addedWithout ParameterStore:Path the app makes no AWS call at start-up
Outside_development_the_app_refuses_to_start_without_a_key_sourceFail fast: no key and no secret means the host does not start in Production
The_configured_setting_selects_the_key_sourcePseudonymisation:Key selects the configuration source; Pseudonymisation:SecretId selects the Secrets Manager source
A_secret_value_is_used_by_the_running_api_and_fetched_once_for_many_requestsThree real requests through the pipeline use the secret's value and cost one GetSecretValue call
Secret_rotation_is_picked_up_after_the_cache_ttl_expiresA new secret version is invisible inside the TTL and visible after it

Create the parameters

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
OWNER=<your-name>
TAGS="Key=Project,Value=cea-lab Key=Lab,Value=m03-accounts-api Key=Environment,Value=learn Key=Owner,Value=$OWNER"

# A secure key. SecureString is encrypted with the AWS managed key alias/aws/ssm (no monthly KMS fee).
aws ssm put-parameter --name /hl-m03/accounts-api/Pseudonymisation/Key --type SecureString \
  --value "$(openssl rand -hex 32)" --tags $TAGS

# A plain setting: the feature flag, off to begin with.
aws ssm put-parameter --name /hl-m03/accounts-api/Features/IncludePendingTransactions --type String \
  --value false --tags $TAGS

aws ssm get-parameters-by-path --path /hl-m03/accounts-api/ --recursive \
  --query "Parameters[].[Name,Type]" --output table

The key's value is not echoed: you did not add --with-decryption. Standard parameters are free; the hierarchy /hl-m03/accounts-api/ is the unit of both configuration and IAM.

Run the API against Parameter Store

export ParameterStore__Path=/hl-m03/accounts-api/
export ParameterStore__ReloadAfter=00:00:20      # lab only: re-read every 20 s (default 5 minutes)
dotnet run --project src/Harbourline.Accounts.Api --launch-profile http

Leave it running. The __ in an environment variable name is how .NET writes the : of ParameterStore:Path. The region comes from AWS_REGION because the Parameter Store provider is created before the app's AWS options exist; if you forget it, start-up fails (see Troubleshooting).

In a second Git Bash window (with the same AWS_PROFILE):

curl -s "http://localhost:5080/v1/accounts/ACC-0000000003/transactions?limit=100" | grep -o '"status":"pending"' | wc -l

With the flag false the count is 0: pending transactions are hidden.

Flip the flag without a restart

aws ssm put-parameter --name /hl-m03/accounts-api/Features/IncludePendingTransactions \
  --type String --value true --overwrite
sleep 25
curl -s "http://localhost:5080/v1/accounts/ACC-0000000003/transactions?limit=100" | grep -o '"status":"pending"' | wc -l

Now the count is 2. No restart, no deployment. Two things made that work: the configuration provider re-reads the path every 20 seconds, and the endpoint reads features.CurrentValue from IOptionsMonitor on each request instead of capturing a value at start-up. Change it back to false when you are done. Note --overwrite and the new parameter version: Parameter Store keeps a history, which is your audit trail of who changed what (together with CloudTrail).

See which layer wins

Stop the API and start it with an environment variable for the same key:

export Features__IncludePendingTransactions=false
dotnet run --project src/Harbourline.Accounts.Api --launch-profile http

Set the parameter to true again and wait: the count becomes 2 even though the environment variable says false, because Parameter Store is a later source than environment variables. Unset the variable (unset Features__IncludePendingTransactions). The lesson is the order, not the flag: when two layers set the same key, the later one wins without any warning. That is how a stale environment variable can hide a corrected parameter, or the reverse, depending on where someone made the change.

Create a secret and read it through the cache

Stop the API. Create the secret and run the API with Pseudonymisation__SecretId and a short cache TTL:

aws secretsmanager create-secret --name hl-m03/pseudonymisation-key \
  --description "Lab 3.5 pseudonymisation key (fictional)" \
  --secret-string "$(openssl rand -hex 32)" --tags $TAGS

export Pseudonymisation__SecretId=hl-m03/pseudonymisation-key
export Pseudonymisation__SecretCacheTtl=00:00:30     # lab only: the default is 15 minutes
dotnet run --project src/Harbourline.Accounts.Api --launch-profile http

SecretId takes precedence over Key, so even though Parameter Store still supplies a Pseudonymisation:Key, the API now uses the secret (the order is in AddPseudonymisation: secret first, then key, then the Development-only fallback).

In the second window:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:5080/v1/customers/CUS-00012345/accounts

Look at the first window: a JSON log line "Listed 2 accounts for customer cref_..." with a CustomerRef property. Note the value. Call the endpoint again and again: the reference is identical (HMAC is deterministic for a key), and only the first call reached Secrets Manager; the rest were served from memory.

Rotate the secret and watch the cache

aws secretsmanager put-secret-value --secret-id hl-m03/pseudonymisation-key \
  --secret-string "$(openssl rand -hex 32)"

# Inside the 30 s TTL: same reference as before.
curl -s -o /dev/null http://localhost:5080/v1/customers/CUS-00012345/accounts
sleep 35
# After the TTL: a different CustomerRef.
curl -s -o /dev/null http://localhost:5080/v1/customers/CUS-00012345/accounts

put-secret-value stores a new version, moves the AWSCURRENT label to it and leaves the previous one labelled AWSPREVIOUS. The API notices only when its cached copy expires and the next read refreshes it. Check the version stages:

aws secretsmanager describe-secret --secret-id hl-m03/pseudonymisation-key \
  --query "VersionIdsToStages"

This is a manual rotation. Real automatic rotation attaches a rotation Lambda to the secret (aws secretsmanager rotate-secret), and is covered by design in Secrets; the application side you just exercised is the same: new AWSCURRENT version, cache refreshes after the TTL.

Think about what rotation just did

Before rotation, every log line about CUS-00012345 carried cref_A. After, the same customer is cref_B. Anyone who correlates logs across the rotation (fraud analysts, support) can no longer join the two periods on that reference. For an ordinary secret that is nothing; for a pseudonymisation key it is a design decision. Write one paragraph answering: which rotation policy do you choose for this key (never, annually, on suspected compromise), how do you keep a key identifier in each log line so old and new references can be told apart, and who is allowed to hold the mapping? There is no single right answer; the Solutions section gives a reasonable one.

Tear down

Stop the API (Ctrl+C), unset the environment variables, and follow Cost and cleanup.

AWS Console instructions

  • Parameter Store: Console → Systems Manager → Application Management → Parameter Store → select a parameter → History tab shows previous versions with the user who changed them. Use Create parameter and pick SecureString to see the KMS key choice.
  • Secrets Manager: Console → Secrets Manager → Secrets → hl-m03/pseudonymisation-key. Overview → Retrieve secret value (this is itself a GetSecretValue call, and shows in CloudTrail). Versions (under Overview/Versions) shows AWSCURRENT and AWSPREVIOUS; Rotation shows the schedule and Lambda, if any.
  • Who read it: Console → CloudTrail → Event history → filter Event name GetSecretValue to see calls made by your SSO session. (Event history takes several minutes to show new events.)

Expected output

# Step 3: flag off
0
# Step 4: after the flag is true and 25 s have passed
2

# API console (compact JSON, abridged)
{"@t":"2026-11-18T10:21:05.1Z","@m":"Listed 2 accounts for customer cref_3f9a07c1d2e4b6a8", ... "CustomerRef":"cref_3f9a07c1d2e4b6a8","CorrelationId":"..."}
# after rotation and the TTL
{"@t":"2026-11-18T10:23:10.4Z","@m":"Listed 2 accounts for customer cref_91c4d0b7e5a2f368", ... "CustomerRef":"cref_91c4d0b7e5a2f368"}

Your references differ because the keys are random. The point is that the first is stable across calls and the second differs.

How the app wires it (and why)

BehaviourWhereWhy
AddSystemsManager(path, optional: false, reloadAfter)HostingExtensions.AddAwsConfigurationSourcesoptional: false means a missing path or a denied call fails start-up loudly rather than running with defaults
Path prefix removed, / becomes :The provider/hl-m03/accounts-api/Pseudonymisation/Key becomes Pseudonymisation:Key
Minimum 5 s reloadParseReloadIntervalA fleet polling every second can exhaust the Parameter Store request quota
Secret first, then key, then Development fallbackAddPseudonymisationThe most managed source wins; a fixed public key exists only in Development
Throw at start-up in Production with no sourceAddPseudonymisationA missing key is found at deployment, not at the first customer request
SecretsManagerCache with CacheItemTTLSecretsManagerKeySourceOne billable call per TTL, not per request; rotation is noticed within the TTL
IOptionsMonitor<FeatureOptions>ListTransactionsAsyncReads the current value per request, so reloads apply without a restart

Infrastructure as Code

The lab uses the CLI so each step is visible. In real projects these resources are code, and the permission that lets a deployed host read them is part of the same change.

An illustrative excerpt (not part of the lab's compiled stacks; try it in the Lambda or Fargate stack and let cdk diff show you the policy it generates):

Excerpt: a parameter, a secret and the grant (illustrative)
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.SSM;

var flag = new StringParameter(this, "IncludePending", new StringParameterProps
{
    ParameterName = "/hl-m03/accounts-api/Features/IncludePendingTransactions",
    StringValue = "false",
});

var key = new Secret(this, "PseudonymisationKey", new SecretProps
{
    SecretName = "hl-m03/pseudonymisation-key",
    GenerateSecretString = new SecretStringGenerator { PasswordLength = 64, ExcludePunctuation = true },
    RemovalPolicy = RemovalPolicy.DESTROY, // ⚠ learning shortcut: production keeps the secret
});

key.GrantRead(function.Role!); // secretsmanager:GetSecretValue and DescribeSecret on this one secret

GrantRead produces exactly the two actions the cache needs on one secret ARN. A secret generated by CloudFormation never passes through your code or your state.

AWS CLI

# Read a parameter (SecureString needs --with-decryption and kms permission for the key)
aws ssm get-parameter --name /hl-m03/accounts-api/Features/IncludePendingTransactions
aws ssm get-parameter-history --name /hl-m03/accounts-api/Features/IncludePendingTransactions \
  --query "Parameters[].[Version,Value,LastModifiedUser]" --output table

# Everything under a path, as the app reads it (metadata only, no values)
aws ssm get-parameters-by-path --path /hl-m03/accounts-api/ --recursive --query "Parameters[].Name"

# Secret metadata and versions (does not return the value)
aws secretsmanager describe-secret --secret-id hl-m03/pseudonymisation-key
aws secretsmanager list-secret-version-ids --secret-id hl-m03/pseudonymisation-key

Complete code

The two pieces that matter are the key sources and the registration that chooses between them.

src/Harbourline.Accounts.Api/Security/Pseudonymisation.cs (excerpt)
public sealed class SecretsManagerKeySource : IPseudonymisationKeySource, IDisposable
{
    private readonly SecretsManagerCache _cache;
    private readonly string _secretId;

    public SecretsManagerKeySource(IAmazonSecretsManager secretsManager, IOptions<PseudonymisationOptions> options)
    {
        _secretId = options.Value.SecretId ?? throw new InvalidOperationException("Pseudonymisation:SecretId is not configured.");
        _cache = new SecretsManagerCache(secretsManager, new SecretCacheConfiguration
        {
            CacheItemTTL = (uint)options.Value.SecretCacheTtl.TotalMilliseconds,
        });
    }

    public async ValueTask<string> GetKeyAsync(CancellationToken cancellationToken) =>
        await _cache.GetSecretString(_secretId, cancellationToken);

    public void Dispose() => _cache.Dispose();
}
src/Harbourline.Accounts.Api/Hosting/HostingExtensions.cs (excerpt)
var parameterPath = builder.Configuration["ParameterStore:Path"];
if (!string.IsNullOrWhiteSpace(parameterPath))
{
    var reload = ParseReloadInterval(builder.Configuration["ParameterStore:ReloadAfter"], DefaultParameterReload);
    builder.Configuration.AddSystemsManager(parameterPath, optional: false, reloadAfter: reload);
}

And the test that proves the cache, with no AWS:

tests/Harbourline.Accounts.Api.Tests/ConfigurationAndLambdaTests.cs (excerpt)
using var source = new SecretsManagerKeySource(secrets, Options.Create(new PseudonymisationOptions
{
    SecretId = "hl-m03/pseudonymisation-key",
    SecretCacheTtl = TimeSpan.FromSeconds(1),
}));

Assert.Equal("secret-key-one", await source.GetKeyAsync(Ct));
rotate("secret-key-two");                                          // like put-secret-value
Assert.Equal("secret-key-one", await source.GetKeyAsync(Ct));    // still inside the TTL: cached
await Task.Delay(TimeSpan.FromMilliseconds(1600), Ct);
Assert.Equal("secret-key-two", await source.GetKeyAsync(Ct));    // after the TTL: refreshed

Code explained line by line

  1. SecretsManagerCache is the official client-side cache. It holds the secret in memory and refreshes it after CacheItemTTL milliseconds; the constructor takes the IAmazonSecretsManager that DI created, so tests can substitute a fake.
  2. (uint)options.Value.SecretCacheTtl.TotalMilliseconds converts the TimeSpan (00:00:30 in the lab, 15 minutes by default) to the milliseconds the cache expects.
  3. GetSecretString(_secretId, ct) returns the AWSCURRENT version's string. On a cold cache it calls DescribeSecret and GetSecretValue; within the TTL it calls neither.
  4. The class is IDisposable and registered as a singleton, so the cache lives for the process and is disposed at shutdown. Never create a cache per request: that defeats it.
  5. ParseReloadInterval accepts a TimeSpan string and rejects anything under five seconds.
  6. optional: false on AddSystemsManager is the fail-fast switch: if the path does not exist or the call is denied, the process exits at start-up.
  7. In the test, rotate(...) changes what the fake returns, exactly as put-secret-value changes the AWSCURRENT version, and the final assertion proves the refresh after the TTL.

Security considerations

⚠ Learning shortcut — not for production

The lab runs the API on your laptop with your SSO permissions (broad in a sandbox), reuses one SecureString key under the AWS managed aws/ssm KMS key, uses a 30 second cache TTL and a 20 second configuration poll to make behaviour visible, and sets recovery_window_in_days = 0 in the Terraform. Production: the host's role with GetParametersByPath on one path and GetSecretValue on one secret ARN, a customer managed KMS key with a key policy that names those roles, TTLs measured in minutes, a recovery window of 7 to 30 days, secret access alarms from CloudTrail, and an automatic rotation schedule agreed with the key's consumers.

  • Least privilege has two halves. The IAM statement in config.tf grants DescribeSecret and GetSecretValue on one ARN. If you encrypt with a customer managed KMS key, the role also needs kms:Decrypt on that key, and the key policy must allow it. A GetParametersByPath permission is evaluated against the path, so the statement's resource must be that path's ARN ⚠ VERIFY the exact resource form for recursive reads against the Systems Manager IAM documentation before copying it into a policy.
  • No secret in environment variables. The lab passes only a secret name in Pseudonymisation__SecretId; the value is fetched at run time and held in memory.
  • No secret in logs. The only derived value written to logs is the HMAC reference. HMAC with a secret key is pseudonymisation: under UK GDPR the reference is still personal data. Whoever holds the key can recompute references for known customer ids, so treat key access as access to personal data and get your data protection officer's view before using this pattern.
  • Anyone who can write the parameter can change behaviour. Restrict ssm:PutParameter on /hl-m03/accounts-api/* as tightly as you restrict deployment, and review changes through CloudTrail and parameter history.
  • A failed secret refresh must not take the API down silently. If Secrets Manager is unreachable when the TTL expires, the next request fails; decide whether to serve the stale value for a short grace period, and alarm on the error.
  • Never claim compliance from a pattern. Encryption, rotation and pseudonymisation are controls inside a larger framework; security and legal review decide whether the design meets your obligations.

Banking use case

Harbourline's fraud team wants support agents to quote a customer reference in tickets without ever seeing the customer id, and wants the logs to remain joinable for ninety days. The platform team stores the HMAC key in Secrets Manager, reads it through a cache with a 15-minute TTL, and puts a key identifier (kid=2026-11) in the reference format so that when the key is rotated at the quarter boundary old and new references remain distinguishable. The feature flag for pending transactions lives in Parameter Store and is changed only through a change record, with the parameter history serving as the audit trail.

Common mistakes

  • Reading Secrets Manager on every request → latency and a bill (USD 0.05 per 10,000 API calls ⚠ VERIFY) → use the cache, singleton.
  • A stale environment variable that is silently overridden or wins → people edit the wrong layer → know the order, and print the effective configuration (without secrets) at start-up in non-production.
  • A flag read once into a field → the reload runs but nothing changes → read IOptionsMonitor.CurrentValue per use.
  • Setting reloadAfter to a few seconds across a large fleet → throttling from Parameter Store → minutes for flags that can wait, AppConfig for faster managed rollout.
  • Putting the secret value in Terraform or CDK source → it lands in state or in the template → let the service generate it, or create it out of band.
  • Rotating a key without asking who depends on it → broken correlation, failed signatures → list consumers first and design for two valid versions during the changeover.
  • Making the Parameter Store provider optional: true for a security setting → the app runs with insecure defaults when the call fails → keep optional: false.

Troubleshooting

SymptomLikely causeFix
App exits: Unable to get IAM security credentials or SSO errorsExpired SSO session or wrong profileaws sso login --profile learn-sandbox; export AWS_PROFILE=learn-sandbox
App exits with a Region or endpoint error from the Parameter Store providerThe provider is created before the app's AWS:Region setting is applied, so the SDK resolves the Region from the environment or profileexport AWS_REGION=us-east-1
AccessDeniedException on GetParametersByPathYour role cannot read the path (or the path in config differs from the real one)Use the same path in ParameterStore__Path and in IAM; both /hl-m03/accounts-api/
App exits: Configure Pseudonymisation:Key ...No key from any source outside DevelopmentCreate the SecureString (step 2), or run with the Development launch profile
Count never changes after flipping the flagWrong parameter name, or the reload has not happenedName must be .../Features/IncludePendingTransactions; wait over 20 s; check the console for provider errors
Count is 2 but should be 0 after setting falseA later layer or a cached valueCheck ParameterStore:ReloadAfter; remember the flag is read per request but the provider reloads on a timer
ResourceNotFoundException for the secretWrong name or Regionaws secretsmanager list-secrets; names are case-sensitive
AccessDeniedException on GetSecretValue, or DescribeSecretThe role has one action but not bothThe cache calls both; grant both on the secret ARN
AccessDenied from KMSSecret encrypted with a customer managed key the role cannot useAdd kms:Decrypt for the role in the key policy
New secret value not seenStill inside the TTLWait the TTL (30 s in the lab), or restart the process
InvalidRequestException: ... scheduled for deletion on re-createSame name reused during the recovery windowRestore the secret, use a new name, or delete with --force-delete-without-recovery in a sandbox

Cost and cleanup

ResourceCost
Parameter Store standard parameters and standard-throughput API callsFree
SecureString with the AWS managed aws/ssm keyNo monthly key fee; requests within the free allowance ⚠ VERIFY
One Secrets Manager secretUSD 0.40 per secret per month, prorated: about USD 0.013 for a day; API calls USD 0.05 per 10,000 (a few dozen calls cost nothing visible)
Local computeFree (your machine)

Label: credits (cents, because of the secret). Free alternative: skip steps 6 and 7 and use only Parameter Store (steps 2 to 5), then read the rotation test in the Complete code section and run the unit tests, which exercise the same cache with a fake.

aws secretsmanager delete-secret --secret-id hl-m03/pseudonymisation-key --force-delete-without-recovery
aws ssm delete-parameters --names \
  /hl-m03/accounts-api/Pseudonymisation/Key \
  /hl-m03/accounts-api/Features/IncludePendingTransactions
unset ParameterStore__Path ParameterStore__ReloadAfter Pseudonymisation__SecretId Pseudonymisation__SecretCacheTtl Features__IncludePendingTransactions

aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m03-accounts-api \
  --query "ResourceTagMappingList[].ResourceARN"

--force-delete-without-recovery skips the recovery window and is for a sandbox. In an account where you want protection, delete with the default window and remember the name stays reserved for that period. If you used Terraform instead of the CLI, terraform destroy removes the same resources.

Exercises

  • 🟢 Guided: Run the API with ParameterStore__Path=/hl-m03/accounts-api/Features/ instead of the parent path. Predict first, then check: is the feature flag still loaded? Is the pseudonymisation key still loaded (remove Pseudonymisation__SecretId for this test)? What does this tell you about the path as a boundary?
  • 🟡 Challenge: Make the pseudonymiser key-versioned: return references of the form cref_<kid>_<hash> where kid is the Secrets Manager VersionId prefix (or a label you store in the secret as JSON). Add tests that two key versions produce different kid values and that references stay stable inside a version.
  • 🔴 Live problem: Security asks you to prove that production's customer-reference key was read only by the Accounts API role in the last 30 days, and to rotate it without breaking the fraud team's ability to search last quarter's logs. Describe the evidence you pull and the rotation procedure you propose.

Assignment

Add CONFIG-AND-SECRETS.md to your portfolio repository: a diagram of your configuration layers, a table of every setting with its layer, whether it is secret and who may change it, the IAM statements a deployed host needs, your decision on rotation for the pseudonymisation key (with the log-correlation trade-off), and a short note on what changes for a bank (customer managed keys, separate accounts for secrets, access alarms).

Solutions

Knowledge check

Interview questions

  • Where would you put a database password, a feature flag and the name of an S3 bucket in an AWS application? Password in Secrets Manager with rotation, read through a cache; the flag in Parameter Store or AppConfig (AppConfig if you need gradual rollout and validation); the bucket name as plain configuration from an environment variable or Parameter Store, because it is not secret.
  • How do you make a running service pick up a changed setting safely? A provider that reloads on a timer, options read through IOptionsMonitor per use, a minimum interval to respect service quotas, and for risky flags a staged rollout with a rollback alarm.
  • You rotate a secret and the application breaks. What do you check? Whether the consumer caches the old value beyond the rotation window, whether both versions are valid during changeover, whether IAM and KMS permissions cover the new version, and whether anything else depends on the old value (signatures, references stored elsewhere).

Certification practice

Completion checklist

  • The tests pass (71 passed, 1 skipped) and I can describe what each configuration test proves
  • I flipped the flag in Parameter Store and saw the API change without a restart
  • I showed which layer won when two layers set the same key
  • I read the secret through the cache, rotated it, and saw the reference change after the TTL
  • I wrote my rotation decision for the pseudonymisation key and its log-correlation trade-off
  • I deleted the secret and parameters and the tag query returned nothing

Key takeaways

  • Configuration is an ordered list of sources; know the order, and make security-relevant sources non-optional.
  • Parameter Store is for settings and flags (hierarchical, free at standard tier); Secrets Manager is for secrets that rotate (a per-secret monthly fee).
  • Cache secrets, read options per use, and keep reload intervals sensible.
  • Rotation changes behaviour for every consumer of the secret, so design the changeover, not just the schedule.
  • The path is the boundary: one path per application and environment, one IAM statement per path.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page