Secrets Manager and Parameter Store
Store, rotate and read secrets the way a bank should: Secrets Manager rotation strategies and resource policies, Parameter Store tiers and SecureString, caching in .NET, the Lambda extension, and how to choose between them.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager 4.1.1.2, AWSSDK.SimpleSystemsManagement 4.1.1.2) · AWSSDK.SecretsManager.Caching 3.0.0 · Amazon.Extensions.Configuration.SystemsManager 7.1.1 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what a secret is, why it must never live in code, configuration files or container images, and what "rotation" really changes.
- Use AWS Secrets Manager: create secrets encrypted with a customer managed KMS key, control access with IAM and resource policies, replicate across Regions, and configure managed rotation or Lambda rotation with the single-user or alternating-users strategy.
- Use Systems Manager Parameter Store: Standard vs Advanced tiers,
SecureStringparameters, hierarchies, and throughput. - Read secrets and configuration from .NET with caching, through the official caching client and the
Amazon.Extensions.Configuration.SystemsManagerconfiguration provider, and from Lambda through the Parameters and Secrets extension. - Choose between Secrets Manager and Parameter Store for a given value, and estimate the cost.
Prerequisites
- AWS KMS — both services encrypt with KMS keys
- IAM in depth — resource-based policies and roles
- Module 3: .NET on AWS —
IConfiguration, dependency injection, ECS/Lambda basics - Tools: AWS CLI v2 with the
learn-sandboxprofile, .NET 10 SDK (or Docker)
Why this matters in banking
Three findings that appear in almost every bank's first cloud penetration test or internal audit:
- A database password in
appsettings.Production.json, committed to Git three years ago. It has been copied to every developer laptop and CI log since, and it has never changed because "changing it would take the payments service down". - A partner API key in a Lambda environment variable, visible to anyone with
lambda:GetFunctionConfiguration— including the read-only support role. - A shared "svc_reporting" database account used by six applications. Nobody can rotate it, because nobody knows all the places it is configured.
Regulators and PCI DSS expect credentials to be protected, access-controlled, logged and changed regularly (PCI DSS v4 has explicit requirements on protecting and periodically changing application and system account passwords; your QSA will interpret them). Secrets Manager and Parameter Store turn those expectations into a design: the secret lives in one place, access is an IAM decision logged in CloudTrail, and rotation is automated so that "changing it" is routine instead of an outage.
The idea in plain language
A secret is any value that grants access: database passwords, API keys, OAuth client secrets, HMAC keys, private keys. The rule is simple: applications fetch secrets at run time from a secret store, using their own identity (an IAM role), and never ship them inside code or images.
- AWS Secrets Manager is a purpose-built secret store. It encrypts each secret with KMS, keeps versions, controls access with IAM and resource policies, and can rotate the secret automatically — generating a new password, setting it in the database, testing it and switching applications over.
- Systems Manager Parameter Store is a hierarchical key/value store for configuration. Values can be plain
String,StringListor encryptedSecureString. It has no built-in rotation, but its Standard tier is free.
Analogy: Secrets Manager is the bank's key safe with a duty officer: every key has a signed log of who took it, and the duty officer changes the locks on a schedule and hands out the new keys. Parameter Store is the branch operations manual: most pages are ordinary instructions anyone on staff may read; a few pages are sealed in envelopes (
SecureString) that only certain people may open.
| Term | Plain meaning |
|---|---|
| Secret version | One value of a secret; versions carry staging labels |
AWSCURRENT | The label on the version applications should use |
AWSPENDING | The new version during a rotation, before it is confirmed |
AWSPREVIOUS | The version that was current before the last rotation (useful for rollback) |
| Rotation | Creating a new credential, setting it in the target system, testing it, and moving AWSCURRENT to it |
| SecureString | A Parameter Store value encrypted with a KMS key |
| Parameter hierarchy | Names like /hl/prod/accounts-api/LogLevel, readable by path |
What problem the AWS services solve
AWS Secrets Manager (checked 2026-11-18)
| Feature | Detail |
|---|---|
| Encryption | Every secret is encrypted with a KMS key: the AWS managed aws/secretsmanager key by default, or a customer managed key (required for cross-account access) |
| Size | Up to 64 KB per secret value ⚠ VERIFY current quota |
| Access control | IAM identity policies plus an optional resource policy on the secret (for cross-account access and for explicit denies) |
| Rotation | Managed rotation for secrets that another service manages (for example RDS/Aurora master passwords created with ManageMasterUserPassword), and rotation by Lambda function for anything else, using AWS templates or your own function; schedules can be as frequent as every four hours ⚠ VERIFY |
| Rotation strategies | Single user (change the password of one account in place) or alternating users (two accounts, rotate the inactive one, then switch) |
| Replication | Replicate a secret to other Regions; replicas stay in sync with the primary for DR |
| Audit | Every GetSecretValue, PutSecretValue and rotation step is in CloudTrail |
| Price | USD 0.40 per secret per month (prorated hourly) and USD 0.05 per 10,000 API calls. New versions created by rotation are not charged separately; the rotation Lambda's own invocations are ordinary Lambda usage |
Systems Manager Parameter Store
| Feature | Standard tier | Advanced tier |
|---|---|---|
| Parameters per account and Region | 10,000 | 100,000 |
| Maximum value size | 4 KB | 8 KB |
| Parameter policies (expiration, notification) | No | Yes |
| Cross-account sharing (through AWS RAM) | No | Yes |
| Cost | No additional charge | Charged per advanced parameter per month and per API interaction ⚠ VERIFY current prices on the Systems Manager pricing page |
| Downgrade | — | Not possible: delete and recreate as Standard |
Throughput: by default Parameter Store serves a modest request rate (40 transactions per second for GetParameter-style calls ⚠ VERIFY); enabling higher throughput for the account and Region raises the limit and makes every API interaction billable. Throughput is a separate setting from the tier.
SecureString parameters are encrypted with the AWS managed key aws/ssm or a customer managed key; readers need ssm:GetParameter and kms:Decrypt on that key.
Choosing between them
| Question | Secrets Manager | Parameter Store |
|---|---|---|
| Does the value need automatic rotation? | ✅ Built in | ❌ Build it yourself |
| Database credentials for RDS/Aurora/Redshift/DocumentDB? | ✅ Templates and managed rotation | Possible, but no rotation |
| Cross-account access to the secret? | ✅ Resource policy + customer managed key | Advanced tier sharing via RAM |
| Cross-Region DR copy? | ✅ Replication | ❌ Copy yourself |
| Thousands of plain configuration values? | Expensive (USD 0.40 each) | ✅ Free (Standard) |
| Feature flags, URLs, log levels | Overkill | ✅ |
| A secret that is never rotated (e.g. a third party's static API key) | ✅ Still best for audit and resource policy | ✅ SecureString is acceptable and free |
Harbourline's rule of thumb: anything that is a credential goes to Secrets Manager; everything else is configuration in Parameter Store. A cheap secret store that is never rotated is still better than a password in appsettings.json.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Key Vault secrets | AWS Secrets Manager | Secrets Manager has built-in rotation workflows (Lambda or managed); Key Vault rotation usually needs Event Grid + Functions |
| Key Vault references in App Service settings | Secrets Manager references through Parameter Store (/aws/reference/secretsmanager/...), ECS task-definition secrets, Lambda extension | ECS injects secrets as environment variables at task start; rotation needs a task restart unless the app reads the secret itself |
| Azure App Configuration | Parameter Store (and AWS AppConfig for feature flags with deployment strategies) | Parameter Store is hierarchical by path; AppConfig adds validation and gradual rollout |
Managed identity + DefaultAzureCredential | IAM role for the task/function + default AWS credential chain | Same idea: no keys in code |
Azure.Extensions.AspNetCore.Configuration.Secrets | Amazon.Extensions.Configuration.SystemsManager | Loads parameters (and referenced secrets) into IConfiguration, with optional reload |
| SQL Server managed identity auth / Entra ID auth | RDS IAM database authentication | Removes the password altogether — worth considering before rotating passwords |
| Key Vault soft delete | Secrets Manager recovery window (7–30 days) | ForceDeleteWithoutRecovery skips it; restrict that action |
Architecture
The rotation sequence for the alternating-users strategy, which avoids the brief window where the old password no longer works:
With the single-user strategy, the same user's password changes in set_secret, so cached copies of the old password fail until the application refreshes. Applications must then retry once with a fresh secret on an authentication error — which is why the reader in the lab has an Invalidate method.
Step-by-step implementation
1. Create a customer managed KMS key for secrets
Use a dedicated key (for example alias/hl-m05-secrets), so the key policy can be tailored to Secrets Manager (kms:ViaService = secretsmanager.<region>.amazonaws.com) and so cross-account sharing is possible later.
2. Create the secret
Name it by owner and purpose, not by environment value: hl-m05/accounts-api/db. Store database credentials as the JSON shape rotation functions expect: username, password, engine, host, port, dbname. Generate the password with GetRandomPassword or let RDS manage it. Tag it.
3. Lock down access
- The application's role gets
secretsmanager:GetSecretValueon that secret's ARN only (ARNs end with a random six-character suffix — usehl-m05/accounts-api/db-??????or the exact ARN), pluskms:Decryptvia Secrets Manager. - Add a resource policy that explicitly denies everyone except the application role, the rotation function and a break-glass role. This protects the secret even if someone is later granted
secretsmanager:*broadly. - Restrict
secretsmanager:DeleteSecretwithForceDeleteWithoutRecoveryandPutResourcePolicyto the platform team.
4. Configure rotation
- RDS/Aurora master user: create the instance with
ManageMasterUserPassword; RDS creates and rotates the secret (managed rotation). - Application users: use the AWS rotation template for your engine (alternating users recommended for high-traffic services), deployed in the database's VPC with access to the Secrets Manager endpoint.
- Start with a 30-day schedule; banks often use 30–90 days for application credentials and shorter for high-risk ones.
5. Put non-secret configuration in Parameter Store
/hl-m05/accounts-api/LogLevel, /hl-m05/accounts-api/Features/RevealRequiresReason. Standard tier, String type. Use SecureString only for low-risk secrets you will not rotate.
6. Read both from .NET with caching
Load configuration once at start-up (with an optional reload interval), and read secrets through a cache with a TTL and an explicit refresh on authentication failure. Never call GetSecretValue per request.
AWS Console instructions
- Console → Secrets Manager → Store a new secret. Choose Other type of secret for a lab (or Credentials for Amazon RDS database for a real database). Enter key/value pairs
usernameandpassword. Choose Encryption keyalias/hl-m05-secretsif you created it, otherwiseaws/secretsmanager. - Secret name:
hl-m05/accounts-api/db. Add the four mandatory tags. Under Resource permissions, choose Edit permissions to add a resource policy. - Configure rotation (optional in the lab): Automatic rotation on, schedule every 30 days or a
rate()/cron()expression, Rotation function: create from a template or choose an existing one. Without a real database, leave rotation off. - Finish, then open the secret: Retrieve secret value shows the value (this is a logged
GetSecretValue). The Versions section showsAWSCURRENT. - Console → Systems Manager → Parameter Store → Create parameter. Name
/hl-m05/accounts-api/LogLevel, Tier Standard, Type String, valueInformation. Create a second one,/hl-m05/accounts-api/PartnerApiKey, Type SecureString, KMS key source current account, keyalias/aws/ssm. - Console → CloudTrail → Event history, filter Event source
secretsmanager.amazonaws.comto see your retrieval.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='[{"Key":"Project","Value":"cea-lab"},{"Key":"Lab","Value":"m05-secrets"},{"Key":"Environment","Value":"learn"},{"Key":"Owner","Value":"alex"}]'
# A strong random password from Secrets Manager itself (no local RNG mistakes).
PW=$(aws secretsmanager get-random-password --password-length 32 --exclude-punctuation \
--query RandomPassword --output text)
# Store database-style credentials (fictional host) with the AWS managed key.
aws secretsmanager create-secret --name hl-m05/accounts-api/db \
--description "Accounts API database credentials (lab, fictional)" \
--secret-string "{\"engine\":\"postgres\",\"host\":\"hl-m05-db.cluster-example.us-east-1.rds.amazonaws.com\",\"port\":5432,\"dbname\":\"accounts\",\"username\":\"accounts_api\",\"password\":\"$PW\"}" \
--tags "$TAGS"
unset PW
# Read it back (this is what the application does) — note it appears in CloudTrail.
aws secretsmanager get-secret-value --secret-id hl-m05/accounts-api/db --query '{Version:VersionId,Stages:VersionStages}'
# Simulate a manual rotation: a new version becomes AWSCURRENT, the old one AWSPREVIOUS.
aws secretsmanager put-secret-value --secret-id hl-m05/accounts-api/db \
--secret-string "{\"engine\":\"postgres\",\"host\":\"hl-m05-db.cluster-example.us-east-1.rds.amazonaws.com\",\"port\":5432,\"dbname\":\"accounts\",\"username\":\"accounts_api\",\"password\":\"$(aws secretsmanager get-random-password --password-length 32 --exclude-punctuation --query RandomPassword --output text)\"}"
aws secretsmanager list-secret-version-ids --secret-id hl-m05/accounts-api/db \
--query 'Versions[].{Id:VersionId,Stages:VersionStages}'
# Parameter Store: plain config and a SecureString.
aws ssm put-parameter --name /hl-m05/accounts-api/LogLevel --type String --value Information \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m05-secrets Key=Environment,Value=learn
aws ssm put-parameter --name /hl-m05/accounts-api/PartnerApiKey --type SecureString \
--value "pk_test_not_a_real_key" --tags Key=Project,Value=cea-lab Key=Lab,Value=m05-secrets Key=Environment,Value=learn
aws ssm get-parameters-by-path --path /hl-m05/accounts-api/ --with-decryption \
--query 'Parameters[].{Name:Name,Type:Type}'
# Secrets Manager values are also reachable through Parameter Store's reference path.
aws ssm get-parameter --name /aws/reference/secretsmanager/hl-m05/accounts-api/db --with-decryption \
--query Parameter.NameInfrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.SSM;
using Constructs;
namespace Harbourline.Security.Infra;
public sealed class SecretsStack : Stack
{
public SecretsStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var key = new Key(this, "SecretsKey", new KeyProps
{
Alias = "alias/hl-m05-cdk-secrets",
EnableKeyRotation = true,
PendingWindow = Duration.Days(7),
RemovalPolicy = RemovalPolicy.DESTROY,
});
// CDK generates the password at deploy time inside Secrets Manager; it never appears in the template.
var secret = new Secret(this, "DbSecret", new SecretProps
{
SecretName = "hl-m05/cdk/accounts-api/db",
EncryptionKey = key,
GenerateSecretString = new SecretStringGenerator
{
SecretStringTemplate = "{\"username\":\"accounts_api\",\"engine\":\"postgres\",\"port\":5432}",
GenerateStringKey = "password",
ExcludePunctuation = true,
PasswordLength = 32,
},
RemovalPolicy = RemovalPolicy.DESTROY, // lab only
});
var apiRole = new Role(this, "ApiRole", new RoleProps
{
AssumedBy = new ServicePrincipal("ecs-tasks.amazonaws.com"),
});
secret.GrantRead(apiRole); // GetSecretValue + DescribeSecret on this secret, kms:Decrypt on the key
// Belt and braces: deny reads to anyone except the API role, even if granted elsewhere.
secret.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.DENY,
Principals = [new AnyPrincipal()],
Actions = ["secretsmanager:GetSecretValue"],
Resources = ["*"],
Conditions = new Dictionary<string, object>
{
["StringNotEquals"] = new Dictionary<string, object> { ["aws:PrincipalArn"] = apiRole.RoleArn },
},
}));
_ = new StringParameter(this, "LogLevel", new StringParameterProps
{
ParameterName = "/hl-m05/cdk/accounts-api/LogLevel",
StringValue = "Information",
Tier = ParameterTier.STANDARD,
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m05-secrets");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
}
}Complete code
An ASP.NET Core minimal API that loads configuration from Parameter Store at start-up and reads database credentials through AWS's caching client, refreshing once on an authentication failure. The lab repository contains an equivalent hand-written cache (CachingSecretReader) with unit tests that run without AWS.
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Secrets.Demo</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Amazon.Extensions.Configuration.SystemsManager" Version="7.1.1" />
<PackageReference Include="AWSSDK.SecretsManager.Caching" Version="3.0.0" />
<PackageReference Include="AWSSDK.SecretsManager" Version="4.1.1.2" />
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
</ItemGroup>
</Project>using System.Text.Json;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Extensions.Caching;
var builder = WebApplication.CreateBuilder(args);
// 1. Non-secret configuration: every parameter under /hl-m05/accounts-api/ becomes a config key
// (/hl-m05/accounts-api/LogLevel → "LogLevel"). Reload every 5 minutes in the background.
builder.Configuration.AddSystemsManager("/hl-m05/accounts-api/", optional: true, reloadAfter: TimeSpan.FromMinutes(5));
// 2. AWS clients from the default credential chain (task role in ECS, SSO profile locally).
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSecretsManager>();
// 3. One cache for the whole process. Values are kept for 15 minutes, then refreshed on next use.
builder.Services.AddSingleton<ISecretsManagerCache>(sp => new SecretsManagerCache(
sp.GetRequiredService<IAmazonSecretsManager>(),
new SecretCacheConfiguration { CacheItemTTL = (uint)TimeSpan.FromMinutes(15).TotalMilliseconds }));
builder.Services.AddSingleton<DbCredentialsProvider>();
var app = builder.Build();
app.MapGet("/config", (IConfiguration config) => Results.Ok(new
{
logLevel = config["LogLevel"] ?? "(not set)",
// Never return secret values from an endpoint. This shows only that the key exists.
partnerKeyConfigured = !string.IsNullOrEmpty(config["PartnerApiKey"]),
}));
app.MapGet("/db-check", async (DbCredentialsProvider credentials, CancellationToken ct) =>
{
var creds = await credentials.GetAsync(ct);
// A real service would open an Npgsql connection here; we only show which user would be used.
return Results.Ok(new { user = creds.Username, host = creds.Host });
});
app.Run();
/// <summary>Database credentials as stored by Secrets Manager. ToString hides the password.</summary>
public sealed record DbCredentials(string Username, string Password, string? Host, int? Port, string? DbName)
{
public override string ToString() => $"DbCredentials {{ Username = {Username}, Password = ***, Host = {Host} }}";
}
public sealed class DbCredentialsProvider(ISecretsManagerCache cache, IConfiguration config, ILogger<DbCredentialsProvider> log)
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
private readonly string _secretId = config["DbSecretId"] ?? "hl-m05/accounts-api/db";
public async Task<DbCredentials> GetAsync(CancellationToken ct = default)
{
var json = await cache.GetSecretString(_secretId, ct);
return JsonSerializer.Deserialize<DbCredentials>(json, Json)
?? throw new InvalidOperationException("Secret is not valid credentials JSON.");
}
/// <summary>
/// Call this when the database rejects the password (e.g. Npgsql PostgresException SqlState 28P01).
/// With single-user rotation the cached password can be stale for up to one TTL.
/// </summary>
public async Task<DbCredentials> RefreshAfterAuthFailureAsync(CancellationToken ct = default)
{
log.LogWarning("Database authentication failed; refreshing secret {SecretId}", _secretId); // the ID, never the value
await cache.RefreshNowAsync(_secretId, ct);
return await GetAsync(ct);
}
}Run it locally against your sandbox:
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws sso login
dotnet run --urls http://localhost:5090
curl -s http://localhost:5090/config
curl -s http://localhost:5090/db-checkReading secrets from Lambda without the SDK cache
The AWS Parameters and Secrets Lambda Extension runs a local cache inside the Lambda execution environment on port 2773. Add the extension layer for your Region and architecture, then:
using System.Net.Http.Json;
public static class LambdaSecretClient
{
private static readonly HttpClient Http = new() { BaseAddress = new Uri("http://localhost:2773") };
public static async Task<string> GetSecretStringAsync(string secretId, CancellationToken ct = default)
{
using var request = new HttpRequestMessage(HttpMethod.Get, $"/secretsmanager/get?secretId={Uri.EscapeDataString(secretId)}");
// The extension only answers callers in the same sandbox that present the session token.
request.Headers.Add("X-Aws-Parameters-Secrets-Token", Environment.GetEnvironmentVariable("AWS_SESSION_TOKEN"));
using var response = await Http.SendAsync(request, ct);
response.EnsureSuccessStatusCode();
var body = await response.Content.ReadFromJsonAsync<ExtensionResponse>(cancellationToken: ct);
return body?.SecretString ?? throw new InvalidOperationException("No SecretString returned.");
}
private sealed record ExtensionResponse(string? SecretString);
}The extension's cache TTL is set with the environment variable SECRETS_MANAGER_TTL (seconds) ⚠ VERIFY the default and maximum in the extension documentation.
Code explained line by line
AddSystemsManager("/hl-m05/accounts-api/", optional: true, reloadAfter: ...)— reads every parameter under the path withGetParametersByPath, strips the prefix and turns/into:, so/hl-m05/accounts-api/Features/RevealbecomesFeatures:Reveal.optional: truelets the app start if the path is empty; in production preferfalseso a missing configuration fails fast.reloadAfter— background reload. Useful for log levels and feature flags; do not rely on it for secrets.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions())+AddAWSService<IAmazonSecretsManager>()— the SDK v4 way to register clients; profile and Region come from theAWSsection or environment.SecretsManagerCache— AWS's caching client (version 3.x targets SDK v4).CacheItemTTLis in milliseconds; one hour is its default. A singleton cache means oneGetSecretValueper secret per TTL per process.- The
/configendpoint returns only whether a secret-like key exists. Never echo secret values in APIs, logs, exceptions or health checks. JsonSerializerDefaults.Webmaps the secret's lower-caseusername/passwordto the record's properties.DbCredentials.ToString()hides the password, so an accidentallog.LogInformation("{Creds}", creds)does not leak it.RefreshNowAsyncforces a reload after an authentication failure. Call it once and retry once — never in a loop, or a genuinely wrong password becomes a self-inflicted throttling incident.- The Lambda client calls the extension on
localhost:2773with theX-Aws-Parameters-Secrets-Tokenheader set to the function's session token; the extension rejects requests without it.
Security considerations
⚠ Learning shortcut — not for production
The lab secret uses the AWS managed key aws/secretsmanager, has no rotation and is readable by your administrator SSO role. In production: a customer managed key with a key policy scoped by kms:ViaService, rotation enabled (managed or Lambda), a resource policy that denies everyone except the application, rotation function and break-glass role, and interface VPC endpoints so secret traffic never leaves the VPC.
- Prefer no secret at all. RDS IAM authentication, IAM roles for AWS APIs and mTLS with short-lived certificates remove whole classes of passwords.
- One secret per application per environment. Shared credentials cannot be rotated safely and destroy audit attribution.
- ECS
secretsin task definitions inject values at task start. After rotation, running tasks keep the old value until restarted — fine with alternating users, risky with single user. Reading through the SDK with a cache avoids this. - Lambda environment variables are not secret storage. Anyone with
lambda:GetFunctionConfigurationcan read them. Use the extension or the SDK. - Do not let Terraform or CloudFormation hold secret values. State files and templates are widely readable. Generate secrets inside Secrets Manager or let the owning service manage them.
- Watch deletions. Restrict
DeleteSecretwithForceDeleteWithoutRecoveryand alert onDeleteSecretandPutResourcePolicyevents. - Logging: log secret IDs and version IDs, never values. Add a test that serialises credential objects and asserts the password is absent (the lab does this).
Banking use case
Harbourline's Accounts API in production:
- Aurora PostgreSQL cluster created with
ManageMasterUserPassword; the master secret is rotated by RDS every 7 days and is readable only by the DBA break-glass role. - Application user
accounts_apiand its cloneaccounts_api_cloneare managed by the alternating-users rotation template every 30 days, running in the database VPC. - The API reads
hl-prod/accounts-api/dbthrough a cache with a 15-minute TTL; on PostgreSQL error28P01(invalid password) it refreshes once and retries. - The blind-index HMAC key for IBAN search (see the module project) lives in
hl-prod/accounts-api/blind-index-key. It is not rotated automatically, because rotating it requires rebuilding the index; a runbook covers manual rotation. - Fifty non-secret settings live under
/hl-prod/accounts-api/in Parameter Store at no charge. - A monthly report from CloudTrail lists every principal that called
GetSecretValueon production secrets; anything other than application roles and the rotation function is investigated.
Cost: 3 secrets × USD 0.40 = USD 1.20 per month, plus API calls (about 4 tasks × 4 refreshes per hour × 730 hours ≈ 11,700 calls ≈ USD 0.06) — trivial compared with the risk removed.
Common mistakes
- Calling
GetSecretValueon every request. → Throttling and cost. Cache in-process with a TTL. - Caching forever. → Rotation breaks the app. Use a TTL and refresh on authentication failure.
- Granting
secretsmanager:GetSecretValueon*. → Any compromised workload can read every secret. Scope to ARNs. - Forgetting
kms:Decryptwith a customer managed key. →AccessDeniedExceptionmentioning KMS. Grant it withkms:ViaService. - Single-user rotation on a busy service without retry logic. → A burst of login failures after every rotation. Use alternating users or add refresh-and-retry.
- Rotation Lambda without network access. → It must reach both the database and the Secrets Manager endpoint (VPC endpoint or NAT). Rotation then fails silently until someone checks.
- Putting thousands of settings in Secrets Manager. → USD 0.40 each; use Parameter Store for configuration.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
AccessDeniedException on GetSecretValue | Missing IAM permission, resource-policy deny, or wildcard ARN without the random suffix | Use the full ARN or name-??????; check the resource policy |
AccessDeniedException ... kms:Decrypt | Customer managed key policy or IAM lacks kms:Decrypt | Grant kms:Decrypt on the key with kms:ViaService |
ResourceNotFoundException | Wrong Region or name; secret scheduled for deletion | Check Region; restore-secret if within the recovery window |
Rotation stuck with AWSPENDING | Rotation function failed at set_secret or test_secret | Read the function's CloudWatch logs; fix network or permissions; re-run rotate-secret |
| App fails for minutes after each rotation | Single-user strategy + long cache TTL, or ECS env-var injection | Alternating users, refresh-on-auth-failure, or restart tasks after rotation |
ThrottlingException from SSM | Default Parameter Store throughput exceeded (often many containers starting at once) | Cache config, use GetParametersByPath once at start-up, or enable higher throughput |
ParameterNotFound with /aws/reference/secretsmanager/... | Reference names are case-sensitive and need --with-decryption | Use the exact secret name and WithDecryption = true |
Cost and cleanup
Cost of this lesson: one secret for an hour costs about USD 0.00056 (USD 0.40 per month prorated hourly); a few dozen API calls cost far less than a cent. Standard parameters are free. If you created a customer managed KMS key, add USD 1 per month prorated (cents for a day).
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Lab only: delete immediately instead of waiting for the recovery window.
aws secretsmanager delete-secret --secret-id hl-m05/accounts-api/db --force-delete-without-recovery
aws ssm delete-parameters --names /hl-m05/accounts-api/LogLevel /hl-m05/accounts-api/PartnerApiKey
# CDK / Terraform versions: cdk destroy | terraform destroy
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m05-secrets
aws secretsmanager list-secrets --filters Key=name,Values=hl-m05 --query 'SecretList[].Name'Exercises
- 🟢 Guided: Run the CLI section, then call
list-secret-version-idsafter theput-secret-valueand explain which version hasAWSCURRENTandAWSPREVIOUS. Find bothGetSecretValuecalls in CloudTrail. - 🟡 Challenge: Write a resource policy for
hl-m05/accounts-api/dbthat deniesGetSecretValueto every principal except one role ARN and any request that does not come through your VPC endpoint (aws:SourceVpce), while still allowing the rotation function. Test the policy logic offline withhl-policy evaluateby writing two scenario files. - 🔴 Live problem: After enabling 30-day single-user rotation on the payments database, the on-call engineer is paged every month at 02:14 with a burst of
password authentication failederrors lasting about 15 minutes. Diagnose and propose two fixes, with trade-offs.
Assignment
Add a secrets folder to your portfolio repository with: a Terraform or CDK module that creates a secret with a customer managed key, a deny-by-default resource policy and optional rotation; a .NET ISecretReader implementation with caching and refresh-on-failure, and unit tests that use a fake Secrets Manager client (see labs/m05-secure-banking-api/tests/Harbourline.SecureBanking.Tests/SecretsTests.cs). Document which values go to Secrets Manager and which to Parameter Store.
Solutions
After put-secret-value, the new version carries AWSCURRENT and the original carries AWSPREVIOUS. Secrets Manager moves the labels automatically when you write a new value without specifying stages. AWSPREVIOUS lets you roll back with update-secret-version-stage if the new credential is wrong. CloudTrail shows GetSecretValue events with requestParameters.secretId and no value — Secrets Manager never logs secret values.
Use one Deny statement with "Principal": "*", "Action": "secretsmanager:GetSecretValue" and two conditions combined by OR — which needs two Deny statements, because conditions inside one statement are ANDed. Statement 1 denies when aws:PrincipalArn is not in [api role, rotation role] (StringNotEquals). Statement 2 denies when aws:SourceVpce is not your endpoint and the principal is the API role (the rotation function may call through a different endpoint). In hl-policy, scenario A (API role, correct endpoint) expects Allow given an identity policy; scenario B (API role, wrong endpoint) expects ExplicitDeny. Watch the missing-key rule: StringNotEquals on an absent aws:SourceVpce (calls not through an endpoint) evaluates to true, so the deny fires — which is what you want.
Cause: single-user rotation changes the password in place at set_secret; running instances keep the old password in their cache for up to the TTL (15 minutes), and new connections fail until each instance refreshes. Fix 1: alternating users — the old user keeps working while the clone gets a new password; no failure window. Costs a second database user and slightly more complex grants. Fix 2: refresh-on-authentication-failure in the application (call RefreshNowAsync on error 28P01, retry once) and shorten the TTL — keeps single user but needs code changes in every client and still produces a few errors. Also schedule rotation in a low-traffic window with a cron() expression and alert on rotation failures, not on the expected brief errors.
Knowledge check
Answer: AWSCURRENT. AWSPENDING marks the new version during rotation, after create_secret and before finish_secret moves AWSCURRENT to it. Applications should not read AWSPENDING.
Answer: A resource policy on the secret allowing the other account's principal, and a customer managed KMS key whose key policy allows that principal to decrypt (the AWS managed aws/secretsmanager key cannot be used cross-account).
Answer: No. Advanced parameters can't be changed back to Standard (values could be truncated, policies would be lost, and the encryption differs). Delete and recreate as Standard.
Answer: Because the credential in use keeps working during rotation: the inactive user gets the new password, then AWSCURRENT switches. Single-user rotation changes the active password and causes failures until every client refreshes.
Answer: Anyone with lambda:GetFunctionConfiguration (and decrypt rights, or the console's automatic decryption with the AWS managed key) can see it, it appears in deployment tooling and IaC, and it cannot be rotated without redeploying. Use Secrets Manager with the Parameters and Secrets extension or the SDK cache.
Interview questions
- "Secrets Manager or Parameter Store?" — Credentials and anything needing rotation, cross-account or cross-Region replication: Secrets Manager. Plain configuration and low-risk static values: Parameter Store Standard, which is free. Mention cost per secret and the rotation strategies.
- "How do you rotate a database password without downtime?" — Alternating-users rotation, applications reading through a cache with a TTL, and refresh-and-retry on authentication failure. Or avoid passwords with IAM database authentication.
- "How would you find out who read a production secret last month?" — CloudTrail
GetSecretValueevents for that secret ID, grouped byuserIdentity.arn; for long-term analysis, query the organisation trail in the log-archive account (Athena or CloudWatch Logs Insights). - "What is wrong with injecting secrets as ECS environment variables?" — Values are fixed at task start, visible to anyone who can exec into or describe the container, and stale after rotation until tasks restart.
Certification practice
A bank must rotate credentials for an Amazon Aurora PostgreSQL application user every 30 days without application errors during rotation. Which solution meets the requirement with the least operational effort?
A. Store the password in Parameter Store as a SecureString and update it with a scheduled script. B. Store the credentials in Secrets Manager and configure rotation with the AWS-provided alternating-users rotation function. C. Store the credentials in Secrets Manager with single-user rotation and restart all application tasks after rotation. D. Use the same database user for all applications and rotate it manually.
Answer: B. Alternating users keeps the active credential valid during rotation and AWS provides the function. A needs custom code and has no rotation workflow. C causes errors and restarts. D is insecure and manual.
An application in account A must read a secret in account B. The secret is encrypted with the aws/secretsmanager key. The resource policy allows account A's role, and the role's IAM policy allows GetSecretValue. Calls fail. What is the fix?
A. Add kms:Decrypt on aws/secretsmanager to the role in account A.
B. Re-encrypt the secret with a customer managed key whose key policy allows account A's role, and grant the role kms:Decrypt on that key.
C. Replicate the secret to account A.
D. Enable rotation on the secret.
Answer: B. AWS managed keys cannot be used cross-account. A cannot work for that reason. C replicates across Regions, not accounts. D is unrelated.
A .NET Lambda function reads a secret on every invocation and is now being throttled at peak, and costs are rising. What should the developer do?
A. Increase the function's memory.
B. Store the secret in a plain environment variable.
C. Use the AWS Parameters and Secrets Lambda Extension (or an in-memory cache with a TTL) to cache the secret across invocations.
D. Call GetSecretValue asynchronously.
Answer: C. Caching reduces calls and cost while respecting rotation through the TTL. A does not reduce calls. B weakens security. D does not reduce the number of calls.
An application with 3,000 non-secret configuration values needs a hierarchical, free-of-charge store readable by path. What should the architect choose?
A. Secrets Manager, one secret per value. B. Systems Manager Parameter Store Standard parameters under a path hierarchy. C. DynamoDB with a GSI. D. S3 objects per value.
Answer: B. Standard parameters are free, hierarchical and readable with GetParametersByPath. A costs USD 1,200 per month. C and D work but add design effort for a solved problem.
Completion checklist
- I can explain
AWSCURRENT,AWSPENDINGandAWSPREVIOUS - I created a secret and a SecureString, read both, and found the reads in CloudTrail
- I can choose single-user vs alternating-users rotation and explain why
- I can read secrets from .NET with a cache and refresh on failure
- I know the price of a secret, an API call and a Standard parameter
- I deleted my lab secret and parameters
Key takeaways
- Credentials go to Secrets Manager (USD 0.40 per secret per month); configuration goes to Parameter Store Standard (free).
- Rotation works only if applications cache with a TTL and refresh on authentication failure; alternating users avoids failures entirely.
- Cross-account secrets need a resource policy and a customer managed KMS key.
- Never put secret values in code, images, environment variables, Terraform state or logs.
- Every secret read is in CloudTrail: use it as your access report.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
When banks need CloudHSM
Decide when a bank genuinely needs single-tenant HSMs: AWS CloudHSM clusters, KMS custom key stores (CloudHSM and external), AWS Payment Cryptography, the PKCS#11 programming model from .NET, and the real operational cost.
Certificates, TLS and mTLS
Issue and auto-renew certificates with ACM, decide when AWS Private CA is worth paying for, pick strict TLS policies on ALB and API Gateway, and build mutual TLS between a partner and a .NET API.