Lab 3.5: Parameter Store and Secrets Manager with caching and rotation
Layer configuration from appsettings, environment variables and Parameter Store, change a feature flag without a restart, read a pseudonymisation key from Secrets Manager through the client-side cache, rotate it, and see exactly when the API notices.
Assumes: .NET 10 LTS · Amazon.Extensions.Configuration.SystemsManager 7.1.1 · AWSSDK.SecretsManager 4.1.1.2 · AWSSDK.SecretsManager.Caching 3.0.0 · Amazon.CDK.Lib 2.277.0 · Terraform 1.14 with AWS provider 6.x · AWS CLI v2 · Region us-east-1 · labs/m03-accounts-api. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Explain the order in which the Accounts API layers configuration, and show which layer won for a given setting.
- Store a plain setting as a String parameter and a key as a SecureString, and load both into
IConfigurationwithParameterStore:Path. - Change a feature flag in Parameter Store and watch a running API pick it up without a restart, using
IOptionsMonitor. - Read a secret from Secrets Manager through the client-side cache, and prove (with a test) that many requests cause one billable call.
- Rotate the secret and describe exactly when the running API sees the new value, and what that does to a pseudonymisation key.
- Write the least-privilege IAM statement a deployed host needs, and tear everything down.
Prerequisites
- Configuration done right and Secrets — the concepts this lab exercises
- AWS clients and dependency injection —
AddAWSServiceand singleton clients - Docker or the .NET 10 SDK; AWS CLI v2 signed in:
aws sso login --profile learn-sandbox - Nothing from Labs 3.3 or 3.4 is needed. This lab runs the API on your machine, using your SSO credentials, so there is no host to deploy.
Why this matters in banking
Two audit questions come up in every review: "where do your applications keep their secrets?" and "how do you change a setting in production, and who can see the change?" A bank needs answers that are boring and repeatable: secrets live in a managed store, encrypted, readable only by the role that needs them, rotated on a schedule; plain settings change through a reviewed process and take effect predictably. This lab builds the application side of both answers, and its last third shows the subtle part nobody warns you about: rotating a key that other systems depend on.
The idea in plain language
An ASP.NET Core app builds its configuration from a list of sources; later sources override earlier ones. The Accounts API's list is:
appsettings.jsonappsettings.{Environment}.json- environment variables
- Parameter Store (only if
ParameterStore:Pathis set) - AppConfig (only if its three ids are set)
Parameter Store is a hierarchical key-value store: a parameter named /hl-m03/accounts-api/Features/IncludePendingTransactions becomes the configuration key Features:IncludePendingTransactions (slashes become colons after the path prefix is removed). A SecureString parameter is encrypted with KMS; the provider decrypts it for you if your role is allowed.
Secrets Manager stores secrets with versions and staging labels (AWSCURRENT, AWSPREVIOUS) and supports rotation. There is no first-party IConfiguration provider for it, so the lab reads it on demand through AWSSDK.SecretsManager.Caching, which keeps the value in memory for a time-to-live (TTL) so the API does not call AWS on every request.
Analogy:
appsettings.jsonis the printed branch handbook. Environment variables are the sticky notes the branch manager adds for this week. Parameter Store is the intranet page head office edits: you re-read it every few minutes. Secrets Manager is the safe: you do not open it for every customer, you take out what you need, keep it in your pocket for a while (the cache), and go back to the safe when the pocket copy is too old.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent in this lab | What's different |
|---|---|---|
appsettings.json + environment + Azure App Configuration provider | AddSystemsManager(path) on builder.Configuration | Same layering idea; the provider polls on a timer (reloadAfter) rather than using a push refresh sentinel |
Key Vault configuration provider (AddAzureKeyVault) | SecretsManagerCache called from a service | No IConfiguration provider: you decide the TTL and the failure behaviour ⚠ VERIFY nothing first-party exists in your package set |
Feature management (IFeatureManager) | IOptionsMonitor<FeatureOptions> over a Parameter Store parameter (simple), or AWS AppConfig feature flags (gradual rollout) | The lab keeps the simple one; the module covers AppConfig |
| Managed identity authorised on the vault | IAM role with ssm:GetParametersByPath / secretsmanager:GetSecretValue on one ARN | Authorisation is an IAM policy on the role, not a vault access policy |
| Key Vault secret versions and rotation policy | Secrets Manager versions, staging labels and rotation Lambda | Rotation is code you (or a template) deploy |
Architecture
Security boundary view. The values never appear in the app's configuration files or its container image. The only thing the process holds is temporary credentials from your SSO profile (and, once deployed, from the host's role). SecureString values and secret values are decrypted in memory and never logged: the API logs a cref_... reference computed from the key, not the key.
What is already in the lab folder
| File | Role in this lab |
|---|---|
src/.../Hosting/HostingExtensions.cs | AddAwsConfigurationSources: Parameter Store with ParameterStore:ReloadAfter (default 5 minutes, minimum 5 seconds), AppConfig when configured |
src/.../Security/Pseudonymisation.cs | PseudonymisationOptions, ConfigurationKeySource, SecretsManagerKeySource (cache) and the HMAC-SHA256 CustomerPseudonymiser |
src/.../Endpoints/AccountEndpoints.cs | Reads IOptionsMonitor<FeatureOptions> per request; logs CustomerRef for GET /v1/customers/{id}/accounts |
tests/.../ConfigurationAndLambdaTests.cs | Tests for the wiring below, using fakes only |
infra/terraform/config.tf | The parameter, the secret container and the IAM statement, as Terraform |
Step-by-step
All commands run from labs/m03-accounts-api in Git Bash.
Run the wiring tests
dotnet test --solution AccountsApi.slnxExpected: 71 passed, 1 skipped. The tests that belong to this lab:
| Test | What it proves |
|---|---|
Parameter_reload_interval_is_parsed_with_a_safe_floor | ParameterStore:ReloadAfter accepts 00:00:20; blanks, nonsense and anything under 5 seconds fall back to 5 minutes |
Without_settings_no_aws_configuration_source_is_added | Without ParameterStore:Path the app makes no AWS call at start-up |
Outside_development_the_app_refuses_to_start_without_a_key_source | Fail fast: no key and no secret means the host does not start in Production |
The_configured_setting_selects_the_key_source | Pseudonymisation:Key selects the configuration source; Pseudonymisation:SecretId selects the Secrets Manager source |
A_secret_value_is_used_by_the_running_api_and_fetched_once_for_many_requests | Three real requests through the pipeline use the secret's value and cost one GetSecretValue call |
Secret_rotation_is_picked_up_after_the_cache_ttl_expires | A new secret version is invisible inside the TTL and visible after it |
Create the parameters
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
OWNER=<your-name>
TAGS="Key=Project,Value=cea-lab Key=Lab,Value=m03-accounts-api Key=Environment,Value=learn Key=Owner,Value=$OWNER"
# A secure key. SecureString is encrypted with the AWS managed key alias/aws/ssm (no monthly KMS fee).
aws ssm put-parameter --name /hl-m03/accounts-api/Pseudonymisation/Key --type SecureString \
--value "$(openssl rand -hex 32)" --tags $TAGS
# A plain setting: the feature flag, off to begin with.
aws ssm put-parameter --name /hl-m03/accounts-api/Features/IncludePendingTransactions --type String \
--value false --tags $TAGS
aws ssm get-parameters-by-path --path /hl-m03/accounts-api/ --recursive \
--query "Parameters[].[Name,Type]" --output tableThe key's value is not echoed: you did not add --with-decryption. Standard parameters are free; the hierarchy /hl-m03/accounts-api/ is the unit of both configuration and IAM.
Run the API against Parameter Store
export ParameterStore__Path=/hl-m03/accounts-api/
export ParameterStore__ReloadAfter=00:00:20 # lab only: re-read every 20 s (default 5 minutes)
dotnet run --project src/Harbourline.Accounts.Api --launch-profile httpLeave it running. The __ in an environment variable name is how .NET writes the : of ParameterStore:Path. The region comes from AWS_REGION because the Parameter Store provider is created before the app's AWS options exist; if you forget it, start-up fails (see Troubleshooting).
In a second Git Bash window (with the same AWS_PROFILE):
curl -s "http://localhost:5080/v1/accounts/ACC-0000000003/transactions?limit=100" | grep -o '"status":"pending"' | wc -lWith the flag false the count is 0: pending transactions are hidden.
Flip the flag without a restart
aws ssm put-parameter --name /hl-m03/accounts-api/Features/IncludePendingTransactions \
--type String --value true --overwrite
sleep 25
curl -s "http://localhost:5080/v1/accounts/ACC-0000000003/transactions?limit=100" | grep -o '"status":"pending"' | wc -lNow the count is 2. No restart, no deployment. Two things made that work: the configuration provider re-reads the path every 20 seconds, and the endpoint reads features.CurrentValue from IOptionsMonitor on each request instead of capturing a value at start-up. Change it back to false when you are done. Note --overwrite and the new parameter version: Parameter Store keeps a history, which is your audit trail of who changed what (together with CloudTrail).
See which layer wins
Stop the API and start it with an environment variable for the same key:
export Features__IncludePendingTransactions=false
dotnet run --project src/Harbourline.Accounts.Api --launch-profile httpSet the parameter to true again and wait: the count becomes 2 even though the environment variable says false, because Parameter Store is a later source than environment variables. Unset the variable (unset Features__IncludePendingTransactions). The lesson is the order, not the flag: when two layers set the same key, the later one wins without any warning. That is how a stale environment variable can hide a corrected parameter, or the reverse, depending on where someone made the change.
Create a secret and read it through the cache
Stop the API. Create the secret and run the API with Pseudonymisation__SecretId and a short cache TTL:
aws secretsmanager create-secret --name hl-m03/pseudonymisation-key \
--description "Lab 3.5 pseudonymisation key (fictional)" \
--secret-string "$(openssl rand -hex 32)" --tags $TAGS
export Pseudonymisation__SecretId=hl-m03/pseudonymisation-key
export Pseudonymisation__SecretCacheTtl=00:00:30 # lab only: the default is 15 minutes
dotnet run --project src/Harbourline.Accounts.Api --launch-profile httpSecretId takes precedence over Key, so even though Parameter Store still supplies a Pseudonymisation:Key, the API now uses the secret (the order is in AddPseudonymisation: secret first, then key, then the Development-only fallback).
In the second window:
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:5080/v1/customers/CUS-00012345/accountsLook at the first window: a JSON log line "Listed 2 accounts for customer cref_..." with a CustomerRef property. Note the value. Call the endpoint again and again: the reference is identical (HMAC is deterministic for a key), and only the first call reached Secrets Manager; the rest were served from memory.
Rotate the secret and watch the cache
aws secretsmanager put-secret-value --secret-id hl-m03/pseudonymisation-key \
--secret-string "$(openssl rand -hex 32)"
# Inside the 30 s TTL: same reference as before.
curl -s -o /dev/null http://localhost:5080/v1/customers/CUS-00012345/accounts
sleep 35
# After the TTL: a different CustomerRef.
curl -s -o /dev/null http://localhost:5080/v1/customers/CUS-00012345/accountsput-secret-value stores a new version, moves the AWSCURRENT label to it and leaves the previous one labelled AWSPREVIOUS. The API notices only when its cached copy expires and the next read refreshes it. Check the version stages:
aws secretsmanager describe-secret --secret-id hl-m03/pseudonymisation-key \
--query "VersionIdsToStages"This is a manual rotation. Real automatic rotation attaches a rotation Lambda to the secret (aws secretsmanager rotate-secret), and is covered by design in Secrets; the application side you just exercised is the same: new AWSCURRENT version, cache refreshes after the TTL.
Think about what rotation just did
Before rotation, every log line about CUS-00012345 carried cref_A. After, the same customer is cref_B. Anyone who correlates logs across the rotation (fraud analysts, support) can no longer join the two periods on that reference. For an ordinary secret that is nothing; for a pseudonymisation key it is a design decision. Write one paragraph answering: which rotation policy do you choose for this key (never, annually, on suspected compromise), how do you keep a key identifier in each log line so old and new references can be told apart, and who is allowed to hold the mapping? There is no single right answer; the Solutions section gives a reasonable one.
Tear down
Stop the API (Ctrl+C), unset the environment variables, and follow Cost and cleanup.
AWS Console instructions
- Parameter Store: Console → Systems Manager → Application Management → Parameter Store → select a parameter → History tab shows previous versions with the user who changed them. Use Create parameter and pick SecureString to see the KMS key choice.
- Secrets Manager: Console → Secrets Manager → Secrets →
hl-m03/pseudonymisation-key. Overview → Retrieve secret value (this is itself aGetSecretValuecall, and shows in CloudTrail). Versions (under Overview/Versions) showsAWSCURRENTandAWSPREVIOUS; Rotation shows the schedule and Lambda, if any. - Who read it: Console → CloudTrail → Event history → filter Event name
GetSecretValueto see calls made by your SSO session. (Event history takes several minutes to show new events.)
Expected output
# Step 3: flag off
0
# Step 4: after the flag is true and 25 s have passed
2
# API console (compact JSON, abridged)
{"@t":"2026-11-18T10:21:05.1Z","@m":"Listed 2 accounts for customer cref_3f9a07c1d2e4b6a8", ... "CustomerRef":"cref_3f9a07c1d2e4b6a8","CorrelationId":"..."}
# after rotation and the TTL
{"@t":"2026-11-18T10:23:10.4Z","@m":"Listed 2 accounts for customer cref_91c4d0b7e5a2f368", ... "CustomerRef":"cref_91c4d0b7e5a2f368"}Your references differ because the keys are random. The point is that the first is stable across calls and the second differs.
How the app wires it (and why)
| Behaviour | Where | Why |
|---|---|---|
AddSystemsManager(path, optional: false, reloadAfter) | HostingExtensions.AddAwsConfigurationSources | optional: false means a missing path or a denied call fails start-up loudly rather than running with defaults |
Path prefix removed, / becomes : | The provider | /hl-m03/accounts-api/Pseudonymisation/Key becomes Pseudonymisation:Key |
| Minimum 5 s reload | ParseReloadInterval | A fleet polling every second can exhaust the Parameter Store request quota |
| Secret first, then key, then Development fallback | AddPseudonymisation | The most managed source wins; a fixed public key exists only in Development |
| Throw at start-up in Production with no source | AddPseudonymisation | A missing key is found at deployment, not at the first customer request |
SecretsManagerCache with CacheItemTTL | SecretsManagerKeySource | One billable call per TTL, not per request; rotation is noticed within the TTL |
IOptionsMonitor<FeatureOptions> | ListTransactionsAsync | Reads the current value per request, so reloads apply without a restart |
Infrastructure as Code
The lab uses the CLI so each step is visible. In real projects these resources are code, and the permission that lets a deployed host read them is part of the same change.
An illustrative excerpt (not part of the lab's compiled stacks; try it in the Lambda or Fargate stack and let cdk diff show you the policy it generates):
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.SSM;
var flag = new StringParameter(this, "IncludePending", new StringParameterProps
{
ParameterName = "/hl-m03/accounts-api/Features/IncludePendingTransactions",
StringValue = "false",
});
var key = new Secret(this, "PseudonymisationKey", new SecretProps
{
SecretName = "hl-m03/pseudonymisation-key",
GenerateSecretString = new SecretStringGenerator { PasswordLength = 64, ExcludePunctuation = true },
RemovalPolicy = RemovalPolicy.DESTROY, // ⚠ learning shortcut: production keeps the secret
});
key.GrantRead(function.Role!); // secretsmanager:GetSecretValue and DescribeSecret on this one secretGrantRead produces exactly the two actions the cache needs on one secret ARN. A secret generated by CloudFormation never passes through your code or your state.
AWS CLI
# Read a parameter (SecureString needs --with-decryption and kms permission for the key)
aws ssm get-parameter --name /hl-m03/accounts-api/Features/IncludePendingTransactions
aws ssm get-parameter-history --name /hl-m03/accounts-api/Features/IncludePendingTransactions \
--query "Parameters[].[Version,Value,LastModifiedUser]" --output table
# Everything under a path, as the app reads it (metadata only, no values)
aws ssm get-parameters-by-path --path /hl-m03/accounts-api/ --recursive --query "Parameters[].Name"
# Secret metadata and versions (does not return the value)
aws secretsmanager describe-secret --secret-id hl-m03/pseudonymisation-key
aws secretsmanager list-secret-version-ids --secret-id hl-m03/pseudonymisation-keyComplete code
The two pieces that matter are the key sources and the registration that chooses between them.
public sealed class SecretsManagerKeySource : IPseudonymisationKeySource, IDisposable
{
private readonly SecretsManagerCache _cache;
private readonly string _secretId;
public SecretsManagerKeySource(IAmazonSecretsManager secretsManager, IOptions<PseudonymisationOptions> options)
{
_secretId = options.Value.SecretId ?? throw new InvalidOperationException("Pseudonymisation:SecretId is not configured.");
_cache = new SecretsManagerCache(secretsManager, new SecretCacheConfiguration
{
CacheItemTTL = (uint)options.Value.SecretCacheTtl.TotalMilliseconds,
});
}
public async ValueTask<string> GetKeyAsync(CancellationToken cancellationToken) =>
await _cache.GetSecretString(_secretId, cancellationToken);
public void Dispose() => _cache.Dispose();
}var parameterPath = builder.Configuration["ParameterStore:Path"];
if (!string.IsNullOrWhiteSpace(parameterPath))
{
var reload = ParseReloadInterval(builder.Configuration["ParameterStore:ReloadAfter"], DefaultParameterReload);
builder.Configuration.AddSystemsManager(parameterPath, optional: false, reloadAfter: reload);
}And the test that proves the cache, with no AWS:
using var source = new SecretsManagerKeySource(secrets, Options.Create(new PseudonymisationOptions
{
SecretId = "hl-m03/pseudonymisation-key",
SecretCacheTtl = TimeSpan.FromSeconds(1),
}));
Assert.Equal("secret-key-one", await source.GetKeyAsync(Ct));
rotate("secret-key-two"); // like put-secret-value
Assert.Equal("secret-key-one", await source.GetKeyAsync(Ct)); // still inside the TTL: cached
await Task.Delay(TimeSpan.FromMilliseconds(1600), Ct);
Assert.Equal("secret-key-two", await source.GetKeyAsync(Ct)); // after the TTL: refreshedCode explained line by line
SecretsManagerCacheis the official client-side cache. It holds the secret in memory and refreshes it afterCacheItemTTLmilliseconds; the constructor takes theIAmazonSecretsManagerthat DI created, so tests can substitute a fake.(uint)options.Value.SecretCacheTtl.TotalMillisecondsconverts the TimeSpan (00:00:30in the lab, 15 minutes by default) to the milliseconds the cache expects.GetSecretString(_secretId, ct)returns theAWSCURRENTversion's string. On a cold cache it callsDescribeSecretandGetSecretValue; within the TTL it calls neither.- The class is
IDisposableand registered as a singleton, so the cache lives for the process and is disposed at shutdown. Never create a cache per request: that defeats it. ParseReloadIntervalaccepts aTimeSpanstring and rejects anything under five seconds.optional: falseonAddSystemsManageris the fail-fast switch: if the path does not exist or the call is denied, the process exits at start-up.- In the test,
rotate(...)changes what the fake returns, exactly asput-secret-valuechanges theAWSCURRENTversion, and the final assertion proves the refresh after the TTL.
Security considerations
⚠ Learning shortcut — not for production
The lab runs the API on your laptop with your SSO permissions (broad in a sandbox), reuses one SecureString key under the AWS managed aws/ssm KMS key, uses a 30 second cache TTL and a 20 second configuration poll to make behaviour visible, and sets recovery_window_in_days = 0 in the Terraform. Production: the host's role with GetParametersByPath on one path and GetSecretValue on one secret ARN, a customer managed KMS key with a key policy that names those roles, TTLs measured in minutes, a recovery window of 7 to 30 days, secret access alarms from CloudTrail, and an automatic rotation schedule agreed with the key's consumers.
- Least privilege has two halves. The IAM statement in
config.tfgrantsDescribeSecretandGetSecretValueon one ARN. If you encrypt with a customer managed KMS key, the role also needskms:Decrypton that key, and the key policy must allow it. AGetParametersByPathpermission is evaluated against the path, so the statement's resource must be that path's ARN ⚠ VERIFY the exact resource form for recursive reads against the Systems Manager IAM documentation before copying it into a policy. - No secret in environment variables. The lab passes only a secret name in
Pseudonymisation__SecretId; the value is fetched at run time and held in memory. - No secret in logs. The only derived value written to logs is the HMAC reference. HMAC with a secret key is pseudonymisation: under UK GDPR the reference is still personal data. Whoever holds the key can recompute references for known customer ids, so treat key access as access to personal data and get your data protection officer's view before using this pattern.
- Anyone who can write the parameter can change behaviour. Restrict
ssm:PutParameteron/hl-m03/accounts-api/*as tightly as you restrict deployment, and review changes through CloudTrail and parameter history. - A failed secret refresh must not take the API down silently. If Secrets Manager is unreachable when the TTL expires, the next request fails; decide whether to serve the stale value for a short grace period, and alarm on the error.
- Never claim compliance from a pattern. Encryption, rotation and pseudonymisation are controls inside a larger framework; security and legal review decide whether the design meets your obligations.
Banking use case
Harbourline's fraud team wants support agents to quote a customer reference in tickets without ever seeing the customer id, and wants the logs to remain joinable for ninety days. The platform team stores the HMAC key in Secrets Manager, reads it through a cache with a 15-minute TTL, and puts a key identifier (kid=2026-11) in the reference format so that when the key is rotated at the quarter boundary old and new references remain distinguishable. The feature flag for pending transactions lives in Parameter Store and is changed only through a change record, with the parameter history serving as the audit trail.
Common mistakes
- Reading Secrets Manager on every request → latency and a bill (USD 0.05 per 10,000 API calls ⚠ VERIFY) → use the cache, singleton.
- A stale environment variable that is silently overridden or wins → people edit the wrong layer → know the order, and print the effective configuration (without secrets) at start-up in non-production.
- A flag read once into a field → the reload runs but nothing changes → read
IOptionsMonitor.CurrentValueper use. - Setting
reloadAfterto a few seconds across a large fleet → throttling from Parameter Store → minutes for flags that can wait, AppConfig for faster managed rollout. - Putting the secret value in Terraform or CDK source → it lands in state or in the template → let the service generate it, or create it out of band.
- Rotating a key without asking who depends on it → broken correlation, failed signatures → list consumers first and design for two valid versions during the changeover.
- Making the Parameter Store provider
optional: truefor a security setting → the app runs with insecure defaults when the call fails → keepoptional: false.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
App exits: Unable to get IAM security credentials or SSO errors | Expired SSO session or wrong profile | aws sso login --profile learn-sandbox; export AWS_PROFILE=learn-sandbox |
| App exits with a Region or endpoint error from the Parameter Store provider | The provider is created before the app's AWS:Region setting is applied, so the SDK resolves the Region from the environment or profile | export AWS_REGION=us-east-1 |
AccessDeniedException on GetParametersByPath | Your role cannot read the path (or the path in config differs from the real one) | Use the same path in ParameterStore__Path and in IAM; both /hl-m03/accounts-api/ |
App exits: Configure Pseudonymisation:Key ... | No key from any source outside Development | Create the SecureString (step 2), or run with the Development launch profile |
| Count never changes after flipping the flag | Wrong parameter name, or the reload has not happened | Name must be .../Features/IncludePendingTransactions; wait over 20 s; check the console for provider errors |
Count is 2 but should be 0 after setting false | A later layer or a cached value | Check ParameterStore:ReloadAfter; remember the flag is read per request but the provider reloads on a timer |
ResourceNotFoundException for the secret | Wrong name or Region | aws secretsmanager list-secrets; names are case-sensitive |
AccessDeniedException on GetSecretValue, or DescribeSecret | The role has one action but not both | The cache calls both; grant both on the secret ARN |
AccessDenied from KMS | Secret encrypted with a customer managed key the role cannot use | Add kms:Decrypt for the role in the key policy |
| New secret value not seen | Still inside the TTL | Wait the TTL (30 s in the lab), or restart the process |
InvalidRequestException: ... scheduled for deletion on re-create | Same name reused during the recovery window | Restore the secret, use a new name, or delete with --force-delete-without-recovery in a sandbox |
Cost and cleanup
| Resource | Cost |
|---|---|
| Parameter Store standard parameters and standard-throughput API calls | Free |
SecureString with the AWS managed aws/ssm key | No monthly key fee; requests within the free allowance ⚠ VERIFY |
| One Secrets Manager secret | USD 0.40 per secret per month, prorated: about USD 0.013 for a day; API calls USD 0.05 per 10,000 (a few dozen calls cost nothing visible) |
| Local compute | Free (your machine) |
Label: credits (cents, because of the secret). Free alternative: skip steps 6 and 7 and use only Parameter Store (steps 2 to 5), then read the rotation test in the Complete code section and run the unit tests, which exercise the same cache with a fake.
aws secretsmanager delete-secret --secret-id hl-m03/pseudonymisation-key --force-delete-without-recovery
aws ssm delete-parameters --names \
/hl-m03/accounts-api/Pseudonymisation/Key \
/hl-m03/accounts-api/Features/IncludePendingTransactions
unset ParameterStore__Path ParameterStore__ReloadAfter Pseudonymisation__SecretId Pseudonymisation__SecretCacheTtl Features__IncludePendingTransactions
aws resourcegroupstaggingapi get-resources \
--tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m03-accounts-api \
--query "ResourceTagMappingList[].ResourceARN"--force-delete-without-recovery skips the recovery window and is for a sandbox. In an account where you want protection, delete with the default window and remember the name stays reserved for that period. If you used Terraform instead of the CLI, terraform destroy removes the same resources.
Exercises
- 🟢 Guided: Run the API with
ParameterStore__Path=/hl-m03/accounts-api/Features/instead of the parent path. Predict first, then check: is the feature flag still loaded? Is the pseudonymisation key still loaded (removePseudonymisation__SecretIdfor this test)? What does this tell you about the path as a boundary? - 🟡 Challenge: Make the pseudonymiser key-versioned: return references of the form
cref_<kid>_<hash>wherekidis the Secrets ManagerVersionIdprefix (or a label you store in the secret as JSON). Add tests that two key versions produce differentkidvalues and that references stay stable inside a version. - 🔴 Live problem: Security asks you to prove that production's customer-reference key was read only by the Accounts API role in the last 30 days, and to rotate it without breaking the fraud team's ability to search last quarter's logs. Describe the evidence you pull and the rotation procedure you propose.
Assignment
Add CONFIG-AND-SECRETS.md to your portfolio repository: a diagram of your configuration layers, a table of every setting with its layer, whether it is secret and who may change it, the IAM statements a deployed host needs, your decision on rotation for the pseudonymisation key (with the log-correlation trade-off), and a short note on what changes for a bank (customer managed keys, separate accounts for secrets, access alarms).
Solutions
With ParameterStore__Path=/hl-m03/accounts-api/Features/ the provider loads only parameters under that prefix, with the prefix removed: the flag arrives as IncludePendingTransactions (not Features:IncludePendingTransactions), so the flag stops working and the app silently uses its default false. Pseudonymisation:Key is not loaded either; in Development the API falls back to the fixed local key (its pseudonymous references differ from the ones you saw before), and in Production it would refuse to start. Two lessons: the prefix you configure is stripped, so it must be the root of your configuration keys; and the path is both the unit of configuration (what the app sees) and the unit of authorisation (what an IAM statement names), so a path per application and environment gives each role only its own settings.
Store the secret as JSON, for example {"kid":"2026-11","key":"..."}, parse it in SecretsManagerKeySource with System.Text.Json, and expose both through an interface such as ValueTask<PseudonymisationKey> GetKeyAsync(...) returning (string Kid, string Key). The pseudonymiser formats cref_{kid}_{hex}. Tests: construct two sources over fakes returning different kid values and assert the prefixes differ; assert the reference for one customer and one version is stable across calls. Why this design: a key identifier travels in the data so a reader of an old log line can tell which key produced it, and rotation never silently changes the meaning of old references.
Evidence: CloudTrail records Secrets Manager API calls such as GetSecretValue and DescribeSecret together with the caller's identity ⚠ VERIFY the event category and your trail's configuration; query the trail (CloudTrail Lake or Athena) for the secret's ARN and group by userIdentity.arn. Anything other than the Accounts API role (a human via SSO, a pipeline) is a finding. Procedure: introduce a new key version with a new kid, have the API use the new key for new references while keeping the old key available to a restricted analytics role for searching old logs, set a retention date for the old key based on your log retention, and rotate on a schedule agreed with fraud, not automatically at 30 days. Record the decision. Note that rotating the secret and re-pseudonymising historical logs are different things: the second is usually not done.
Knowledge check
Answer: appsettings.json, then environment variables, then Parameter Store; later wins, so Parameter Store overrides both. Moving a setting between layers therefore changes which value is effective.
Answer: The provider re-reads the path on a timer (reloadAfter), and the endpoint reads IOptionsMonitor.CurrentValue on each request. Capturing the value in a field or a singleton at start-up would freeze it, as would a reload interval longer than you expect or an earlier-win mistake in the layering.
Answer: Each call costs money and adds latency, and a busy API would hit API limits. The cache holds the value for a TTL, so the number of calls is one per TTL per process, while still picking up a rotated value within that time.
Answer: The new version, which carries the AWSCURRENT label. The API reads it after its cached copy expires (up to the TTL later) and the next request refreshes it. Until then it keeps using the old value.
Answer: The caching client first asks which version is current (DescribeSecret) and then fetches that version (GetSecretValue). Granting only the second makes the cache fail with access denied on the first call.
Interview questions
- Where would you put a database password, a feature flag and the name of an S3 bucket in an AWS application? Password in Secrets Manager with rotation, read through a cache; the flag in Parameter Store or AppConfig (AppConfig if you need gradual rollout and validation); the bucket name as plain configuration from an environment variable or Parameter Store, because it is not secret.
- How do you make a running service pick up a changed setting safely? A provider that reloads on a timer, options read through
IOptionsMonitorper use, a minimum interval to respect service quotas, and for risky flags a staged rollout with a rollback alarm. - You rotate a secret and the application breaks. What do you check? Whether the consumer caches the old value beyond the rotation window, whether both versions are valid during changeover, whether IAM and KMS permissions cover the new version, and whether anything else depends on the old value (signatures, references stored elsewhere).
Certification practice
An application running on ECS needs a database password that rotates every 30 days. The application must not store the password in its image or in task definition environment variables. What is the best solution? A. Store the password in Secrets Manager and retrieve it at run time with the task role. B. Put it in an S3 object and download it at start-up. C. Put it in a Parameter Store String parameter. D. Encode it in Base64 in the task definition. Answer: A. Secrets Manager is built for rotating credentials and the task role gives short-lived access. B and C do not provide rotation or encryption in the way required (a String parameter is not encrypted); D is encoding, not protection.
A high-traffic API retrieves a secret from Secrets Manager on every request and sees throttling and rising cost. Which change fixes this with the least code? A. Increase the function memory. B. Cache the secret in memory with a time-to-live, using the Secrets Manager caching client. C. Store the secret in an environment variable. D. Call GetSecretValue from a bigger instance.
Answer: B. The caching component reduces calls to one per TTL and still picks up rotation. A and D change capacity, not the call pattern. C avoids the calls but exposes the value in configuration and breaks rotation.
A company stores an API key as a Parameter Store parameter and must keep it encrypted at rest and limit who can decrypt it to one application role. Which approach meets this? A. A String parameter with a restrictive resource policy. B. A SecureString parameter encrypted with a customer managed KMS key whose key policy allows only that role. C. A StringList parameter. D. A tag on the parameter marking it confidential. Answer: B. SecureString encrypts the value with KMS, and a customer managed key lets you control decrypt access in the key policy. A leaves the value in plain text; C is a list type and not encrypted; D is a label and enforces nothing.
Completion checklist
- The tests pass (71 passed, 1 skipped) and I can describe what each configuration test proves
- I flipped the flag in Parameter Store and saw the API change without a restart
- I showed which layer won when two layers set the same key
- I read the secret through the cache, rotated it, and saw the reference change after the TTL
- I wrote my rotation decision for the pseudonymisation key and its log-correlation trade-off
- I deleted the secret and parameters and the tag query returned nothing
Key takeaways
- Configuration is an ordered list of sources; know the order, and make security-relevant sources non-optional.
- Parameter Store is for settings and flags (hierarchical, free at standard tier); Secrets Manager is for secrets that rotate (a per-secret monthly fee).
- Cache secrets, read options per use, and keep reload intervals sensible.
- Rotation changes behaviour for every consumer of the secret, so design the changeover, not just the schedule.
- The path is the boundary: one path per application and environment, one IAM statement per path.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 3.4: run the Accounts API on Lambda (dotnet10) behind an HTTP API
Publish the same ASP.NET Core Accounts API for linux-arm64, deploy it to the Lambda dotnet10 runtime behind an API Gateway HTTP API, replay a real HTTP API event, read cold starts from the REPORT lines, and compare memory sizes, ReadyToRun and SnapStart.
Scaling .NET apps on AWS
Make an ASP.NET Core API stateless, size its connection pools, move session state to ElastiCache or DynamoDB, and scale it automatically with target tracking on ECS and concurrency controls on Lambda.