Skip to Content
Cloud Engineering Academy
5. Security for Banking & Finance

Secrets Manager and Parameter Store

Store, rotate and read secrets the way a bank should: Secrets Manager rotation strategies and resource policies, Parameter Store tiers and SecureString, caching in .NET, the Lambda extension, and how to choose between them.

intermediate⏱ 2 h 🟡 Low costSCS-C03DVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager 4.1.1.2, AWSSDK.SimpleSystemsManagement 4.1.1.2) · AWSSDK.SecretsManager.Caching 3.0.0 · Amazon.Extensions.Configuration.SystemsManager 7.1.1 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what a secret is, why it must never live in code, configuration files or container images, and what "rotation" really changes.
  • Use AWS Secrets Manager: create secrets encrypted with a customer managed KMS key, control access with IAM and resource policies, replicate across Regions, and configure managed rotation or Lambda rotation with the single-user or alternating-users strategy.
  • Use Systems Manager Parameter Store: Standard vs Advanced tiers, SecureString parameters, hierarchies, and throughput.
  • Read secrets and configuration from .NET with caching, through the official caching client and the Amazon.Extensions.Configuration.SystemsManager configuration provider, and from Lambda through the Parameters and Secrets extension.
  • Choose between Secrets Manager and Parameter Store for a given value, and estimate the cost.

Prerequisites

  • AWS KMS — both services encrypt with KMS keys
  • IAM in depth — resource-based policies and roles
  • Module 3: .NET on AWS — IConfiguration, dependency injection, ECS/Lambda basics
  • Tools: AWS CLI v2 with the learn-sandbox profile, .NET 10 SDK (or Docker)

Why this matters in banking

Three findings that appear in almost every bank's first cloud penetration test or internal audit:

  1. A database password in appsettings.Production.json, committed to Git three years ago. It has been copied to every developer laptop and CI log since, and it has never changed because "changing it would take the payments service down".
  2. A partner API key in a Lambda environment variable, visible to anyone with lambda:GetFunctionConfiguration — including the read-only support role.
  3. A shared "svc_reporting" database account used by six applications. Nobody can rotate it, because nobody knows all the places it is configured.

Regulators and PCI DSS expect credentials to be protected, access-controlled, logged and changed regularly (PCI DSS v4 has explicit requirements on protecting and periodically changing application and system account passwords; your QSA will interpret them). Secrets Manager and Parameter Store turn those expectations into a design: the secret lives in one place, access is an IAM decision logged in CloudTrail, and rotation is automated so that "changing it" is routine instead of an outage.

The idea in plain language

A secret is any value that grants access: database passwords, API keys, OAuth client secrets, HMAC keys, private keys. The rule is simple: applications fetch secrets at run time from a secret store, using their own identity (an IAM role), and never ship them inside code or images.

  • AWS Secrets Manager is a purpose-built secret store. It encrypts each secret with KMS, keeps versions, controls access with IAM and resource policies, and can rotate the secret automatically — generating a new password, setting it in the database, testing it and switching applications over.
  • Systems Manager Parameter Store is a hierarchical key/value store for configuration. Values can be plain String, StringList or encrypted SecureString. It has no built-in rotation, but its Standard tier is free.

Analogy: Secrets Manager is the bank's key safe with a duty officer: every key has a signed log of who took it, and the duty officer changes the locks on a schedule and hands out the new keys. Parameter Store is the branch operations manual: most pages are ordinary instructions anyone on staff may read; a few pages are sealed in envelopes (SecureString) that only certain people may open.

TermPlain meaning
Secret versionOne value of a secret; versions carry staging labels
AWSCURRENTThe label on the version applications should use
AWSPENDINGThe new version during a rotation, before it is confirmed
AWSPREVIOUSThe version that was current before the last rotation (useful for rollback)
RotationCreating a new credential, setting it in the target system, testing it, and moving AWSCURRENT to it
SecureStringA Parameter Store value encrypted with a KMS key
Parameter hierarchyNames like /hl/prod/accounts-api/LogLevel, readable by path

What problem the AWS services solve

AWS Secrets Manager (checked 2026-11-18)

FeatureDetail
EncryptionEvery secret is encrypted with a KMS key: the AWS managed aws/secretsmanager key by default, or a customer managed key (required for cross-account access)
SizeUp to 64 KB per secret value ⚠ VERIFY current quota
Access controlIAM identity policies plus an optional resource policy on the secret (for cross-account access and for explicit denies)
RotationManaged rotation for secrets that another service manages (for example RDS/Aurora master passwords created with ManageMasterUserPassword), and rotation by Lambda function for anything else, using AWS templates or your own function; schedules can be as frequent as every four hours ⚠ VERIFY
Rotation strategiesSingle user (change the password of one account in place) or alternating users (two accounts, rotate the inactive one, then switch)
ReplicationReplicate a secret to other Regions; replicas stay in sync with the primary for DR
AuditEvery GetSecretValue, PutSecretValue and rotation step is in CloudTrail
PriceUSD 0.40 per secret per month (prorated hourly) and USD 0.05 per 10,000 API calls. New versions created by rotation are not charged separately; the rotation Lambda's own invocations are ordinary Lambda usage

Systems Manager Parameter Store

FeatureStandard tierAdvanced tier
Parameters per account and Region10,000100,000
Maximum value size4 KB8 KB
Parameter policies (expiration, notification)NoYes
Cross-account sharing (through AWS RAM)NoYes
CostNo additional chargeCharged per advanced parameter per month and per API interaction ⚠ VERIFY current prices on the Systems Manager pricing page
Downgrade—Not possible: delete and recreate as Standard

Throughput: by default Parameter Store serves a modest request rate (40 transactions per second for GetParameter-style calls ⚠ VERIFY); enabling higher throughput for the account and Region raises the limit and makes every API interaction billable. Throughput is a separate setting from the tier.

SecureString parameters are encrypted with the AWS managed key aws/ssm or a customer managed key; readers need ssm:GetParameter and kms:Decrypt on that key.

Choosing between them

QuestionSecrets ManagerParameter Store
Does the value need automatic rotation?✅ Built in❌ Build it yourself
Database credentials for RDS/Aurora/Redshift/DocumentDB?✅ Templates and managed rotationPossible, but no rotation
Cross-account access to the secret?✅ Resource policy + customer managed keyAdvanced tier sharing via RAM
Cross-Region DR copy?✅ Replication❌ Copy yourself
Thousands of plain configuration values?Expensive (USD 0.40 each)✅ Free (Standard)
Feature flags, URLs, log levelsOverkill✅
A secret that is never rotated (e.g. a third party's static API key)✅ Still best for audit and resource policy✅ SecureString is acceptable and free

Harbourline's rule of thumb: anything that is a credential goes to Secrets Manager; everything else is configuration in Parameter Store. A cheap secret store that is never rotated is still better than a password in appsettings.json.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Key Vault secretsAWS Secrets ManagerSecrets Manager has built-in rotation workflows (Lambda or managed); Key Vault rotation usually needs Event Grid + Functions
Key Vault references in App Service settingsSecrets Manager references through Parameter Store (/aws/reference/secretsmanager/...), ECS task-definition secrets, Lambda extensionECS injects secrets as environment variables at task start; rotation needs a task restart unless the app reads the secret itself
Azure App ConfigurationParameter Store (and AWS AppConfig for feature flags with deployment strategies)Parameter Store is hierarchical by path; AppConfig adds validation and gradual rollout
Managed identity + DefaultAzureCredentialIAM role for the task/function + default AWS credential chainSame idea: no keys in code
Azure.Extensions.AspNetCore.Configuration.SecretsAmazon.Extensions.Configuration.SystemsManagerLoads parameters (and referenced secrets) into IConfiguration, with optional reload
SQL Server managed identity auth / Entra ID authRDS IAM database authenticationRemoves the password altogether — worth considering before rotating passwords
Key Vault soft deleteSecrets Manager recovery window (7–30 days)ForceDeleteWithoutRecovery skips it; restrict that action

Architecture

The rotation sequence for the alternating-users strategy, which avoids the brief window where the old password no longer works:

With the single-user strategy, the same user's password changes in set_secret, so cached copies of the old password fail until the application refreshes. Applications must then retry once with a fresh secret on an authentication error — which is why the reader in the lab has an Invalidate method.

Step-by-step implementation

1. Create a customer managed KMS key for secrets

Use a dedicated key (for example alias/hl-m05-secrets), so the key policy can be tailored to Secrets Manager (kms:ViaService = secretsmanager.<region>.amazonaws.com) and so cross-account sharing is possible later.

2. Create the secret

Name it by owner and purpose, not by environment value: hl-m05/accounts-api/db. Store database credentials as the JSON shape rotation functions expect: username, password, engine, host, port, dbname. Generate the password with GetRandomPassword or let RDS manage it. Tag it.

3. Lock down access

  • The application's role gets secretsmanager:GetSecretValue on that secret's ARN only (ARNs end with a random six-character suffix — use hl-m05/accounts-api/db-?????? or the exact ARN), plus kms:Decrypt via Secrets Manager.
  • Add a resource policy that explicitly denies everyone except the application role, the rotation function and a break-glass role. This protects the secret even if someone is later granted secretsmanager:* broadly.
  • Restrict secretsmanager:DeleteSecret with ForceDeleteWithoutRecovery and PutResourcePolicy to the platform team.

4. Configure rotation

  • RDS/Aurora master user: create the instance with ManageMasterUserPassword; RDS creates and rotates the secret (managed rotation).
  • Application users: use the AWS rotation template for your engine (alternating users recommended for high-traffic services), deployed in the database's VPC with access to the Secrets Manager endpoint.
  • Start with a 30-day schedule; banks often use 30–90 days for application credentials and shorter for high-risk ones.

5. Put non-secret configuration in Parameter Store

/hl-m05/accounts-api/LogLevel, /hl-m05/accounts-api/Features/RevealRequiresReason. Standard tier, String type. Use SecureString only for low-risk secrets you will not rotate.

6. Read both from .NET with caching

Load configuration once at start-up (with an optional reload interval), and read secrets through a cache with a TTL and an explicit refresh on authentication failure. Never call GetSecretValue per request.

AWS Console instructions

  1. Console → Secrets Manager → Store a new secret. Choose Other type of secret for a lab (or Credentials for Amazon RDS database for a real database). Enter key/value pairs username and password. Choose Encryption key alias/hl-m05-secrets if you created it, otherwise aws/secretsmanager.
  2. Secret name: hl-m05/accounts-api/db. Add the four mandatory tags. Under Resource permissions, choose Edit permissions to add a resource policy.
  3. Configure rotation (optional in the lab): Automatic rotation on, schedule every 30 days or a rate()/cron() expression, Rotation function: create from a template or choose an existing one. Without a real database, leave rotation off.
  4. Finish, then open the secret: Retrieve secret value shows the value (this is a logged GetSecretValue). The Versions section shows AWSCURRENT.
  5. Console → Systems Manager → Parameter Store → Create parameter. Name /hl-m05/accounts-api/LogLevel, Tier Standard, Type String, value Information. Create a second one, /hl-m05/accounts-api/PartnerApiKey, Type SecureString, KMS key source current account, key alias/aws/ssm.
  6. Console → CloudTrail → Event history, filter Event source secretsmanager.amazonaws.com to see your retrieval.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='[{"Key":"Project","Value":"cea-lab"},{"Key":"Lab","Value":"m05-secrets"},{"Key":"Environment","Value":"learn"},{"Key":"Owner","Value":"alex"}]'

# A strong random password from Secrets Manager itself (no local RNG mistakes).
PW=$(aws secretsmanager get-random-password --password-length 32 --exclude-punctuation \
     --query RandomPassword --output text)

# Store database-style credentials (fictional host) with the AWS managed key.
aws secretsmanager create-secret --name hl-m05/accounts-api/db \
  --description "Accounts API database credentials (lab, fictional)" \
  --secret-string "{\"engine\":\"postgres\",\"host\":\"hl-m05-db.cluster-example.us-east-1.rds.amazonaws.com\",\"port\":5432,\"dbname\":\"accounts\",\"username\":\"accounts_api\",\"password\":\"$PW\"}" \
  --tags "$TAGS"
unset PW

# Read it back (this is what the application does) — note it appears in CloudTrail.
aws secretsmanager get-secret-value --secret-id hl-m05/accounts-api/db --query '{Version:VersionId,Stages:VersionStages}'

# Simulate a manual rotation: a new version becomes AWSCURRENT, the old one AWSPREVIOUS.
aws secretsmanager put-secret-value --secret-id hl-m05/accounts-api/db \
  --secret-string "{\"engine\":\"postgres\",\"host\":\"hl-m05-db.cluster-example.us-east-1.rds.amazonaws.com\",\"port\":5432,\"dbname\":\"accounts\",\"username\":\"accounts_api\",\"password\":\"$(aws secretsmanager get-random-password --password-length 32 --exclude-punctuation --query RandomPassword --output text)\"}"
aws secretsmanager list-secret-version-ids --secret-id hl-m05/accounts-api/db \
  --query 'Versions[].{Id:VersionId,Stages:VersionStages}'

# Parameter Store: plain config and a SecureString.
aws ssm put-parameter --name /hl-m05/accounts-api/LogLevel --type String --value Information \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m05-secrets Key=Environment,Value=learn
aws ssm put-parameter --name /hl-m05/accounts-api/PartnerApiKey --type SecureString \
  --value "pk_test_not_a_real_key" --tags Key=Project,Value=cea-lab Key=Lab,Value=m05-secrets Key=Environment,Value=learn
aws ssm get-parameters-by-path --path /hl-m05/accounts-api/ --with-decryption \
  --query 'Parameters[].{Name:Name,Type:Type}'

# Secrets Manager values are also reachable through Parameter Store's reference path.
aws ssm get-parameter --name /aws/reference/secretsmanager/hl-m05/accounts-api/db --with-decryption \
  --query Parameter.Name

Infrastructure as Code

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.SSM;
using Constructs;

namespace Harbourline.Security.Infra;

public sealed class SecretsStack : Stack
{
    public SecretsStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var key = new Key(this, "SecretsKey", new KeyProps
        {
            Alias = "alias/hl-m05-cdk-secrets",
            EnableKeyRotation = true,
            PendingWindow = Duration.Days(7),
            RemovalPolicy = RemovalPolicy.DESTROY,
        });

        // CDK generates the password at deploy time inside Secrets Manager; it never appears in the template.
        var secret = new Secret(this, "DbSecret", new SecretProps
        {
            SecretName = "hl-m05/cdk/accounts-api/db",
            EncryptionKey = key,
            GenerateSecretString = new SecretStringGenerator
            {
                SecretStringTemplate = "{\"username\":\"accounts_api\",\"engine\":\"postgres\",\"port\":5432}",
                GenerateStringKey = "password",
                ExcludePunctuation = true,
                PasswordLength = 32,
            },
            RemovalPolicy = RemovalPolicy.DESTROY, // lab only
        });

        var apiRole = new Role(this, "ApiRole", new RoleProps
        {
            AssumedBy = new ServicePrincipal("ecs-tasks.amazonaws.com"),
        });
        secret.GrantRead(apiRole); // GetSecretValue + DescribeSecret on this secret, kms:Decrypt on the key

        // Belt and braces: deny reads to anyone except the API role, even if granted elsewhere.
        secret.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Effect = Effect.DENY,
            Principals = [new AnyPrincipal()],
            Actions = ["secretsmanager:GetSecretValue"],
            Resources = ["*"],
            Conditions = new Dictionary<string, object>
            {
                ["StringNotEquals"] = new Dictionary<string, object> { ["aws:PrincipalArn"] = apiRole.RoleArn },
            },
        }));

        _ = new StringParameter(this, "LogLevel", new StringParameterProps
        {
            ParameterName = "/hl-m05/cdk/accounts-api/LogLevel",
            StringValue = "Information",
            Tier = ParameterTier.STANDARD,
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m05-secrets");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
    }
}

Complete code

An ASP.NET Core minimal API that loads configuration from Parameter Store at start-up and reads database credentials through AWS's caching client, refreshing once on an authentication failure. The lab repository contains an equivalent hand-written cache (CachingSecretReader) with unit tests that run without AWS.

Harbourline.Secrets.Demo.csproj
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Secrets.Demo</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Amazon.Extensions.Configuration.SystemsManager" Version="7.1.1" />
    <PackageReference Include="AWSSDK.SecretsManager.Caching" Version="3.0.0" />
    <PackageReference Include="AWSSDK.SecretsManager" Version="4.1.1.2" />
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
  </ItemGroup>
</Project>
Program.cs
using System.Text.Json;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Extensions.Caching;

var builder = WebApplication.CreateBuilder(args);

// 1. Non-secret configuration: every parameter under /hl-m05/accounts-api/ becomes a config key
//    (/hl-m05/accounts-api/LogLevel → "LogLevel"). Reload every 5 minutes in the background.
builder.Configuration.AddSystemsManager("/hl-m05/accounts-api/", optional: true, reloadAfter: TimeSpan.FromMinutes(5));

// 2. AWS clients from the default credential chain (task role in ECS, SSO profile locally).
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSecretsManager>();

// 3. One cache for the whole process. Values are kept for 15 minutes, then refreshed on next use.
builder.Services.AddSingleton<ISecretsManagerCache>(sp => new SecretsManagerCache(
    sp.GetRequiredService<IAmazonSecretsManager>(),
    new SecretCacheConfiguration { CacheItemTTL = (uint)TimeSpan.FromMinutes(15).TotalMilliseconds }));
builder.Services.AddSingleton<DbCredentialsProvider>();

var app = builder.Build();

app.MapGet("/config", (IConfiguration config) => Results.Ok(new
{
    logLevel = config["LogLevel"] ?? "(not set)",
    // Never return secret values from an endpoint. This shows only that the key exists.
    partnerKeyConfigured = !string.IsNullOrEmpty(config["PartnerApiKey"]),
}));

app.MapGet("/db-check", async (DbCredentialsProvider credentials, CancellationToken ct) =>
{
    var creds = await credentials.GetAsync(ct);
    // A real service would open an Npgsql connection here; we only show which user would be used.
    return Results.Ok(new { user = creds.Username, host = creds.Host });
});

app.Run();

/// <summary>Database credentials as stored by Secrets Manager. ToString hides the password.</summary>
public sealed record DbCredentials(string Username, string Password, string? Host, int? Port, string? DbName)
{
    public override string ToString() => $"DbCredentials {{ Username = {Username}, Password = ***, Host = {Host} }}";
}

public sealed class DbCredentialsProvider(ISecretsManagerCache cache, IConfiguration config, ILogger<DbCredentialsProvider> log)
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
    private readonly string _secretId = config["DbSecretId"] ?? "hl-m05/accounts-api/db";

    public async Task<DbCredentials> GetAsync(CancellationToken ct = default)
    {
        var json = await cache.GetSecretString(_secretId, ct);
        return JsonSerializer.Deserialize<DbCredentials>(json, Json)
            ?? throw new InvalidOperationException("Secret is not valid credentials JSON.");
    }

    /// <summary>
    /// Call this when the database rejects the password (e.g. Npgsql PostgresException SqlState 28P01).
    /// With single-user rotation the cached password can be stale for up to one TTL.
    /// </summary>
    public async Task<DbCredentials> RefreshAfterAuthFailureAsync(CancellationToken ct = default)
    {
        log.LogWarning("Database authentication failed; refreshing secret {SecretId}", _secretId); // the ID, never the value
        await cache.RefreshNowAsync(_secretId, ct);
        return await GetAsync(ct);
    }
}

Run it locally against your sandbox:

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws sso login
dotnet run --urls http://localhost:5090
curl -s http://localhost:5090/config
curl -s http://localhost:5090/db-check

Reading secrets from Lambda without the SDK cache

The AWS Parameters and Secrets Lambda Extension runs a local cache inside the Lambda execution environment on port 2773. Add the extension layer for your Region and architecture, then:

LambdaSecretClient.cs
using System.Net.Http.Json;

public static class LambdaSecretClient
{
    private static readonly HttpClient Http = new() { BaseAddress = new Uri("http://localhost:2773") };

    public static async Task<string> GetSecretStringAsync(string secretId, CancellationToken ct = default)
    {
        using var request = new HttpRequestMessage(HttpMethod.Get, $"/secretsmanager/get?secretId={Uri.EscapeDataString(secretId)}");
        // The extension only answers callers in the same sandbox that present the session token.
        request.Headers.Add("X-Aws-Parameters-Secrets-Token", Environment.GetEnvironmentVariable("AWS_SESSION_TOKEN"));
        using var response = await Http.SendAsync(request, ct);
        response.EnsureSuccessStatusCode();
        var body = await response.Content.ReadFromJsonAsync<ExtensionResponse>(cancellationToken: ct);
        return body?.SecretString ?? throw new InvalidOperationException("No SecretString returned.");
    }

    private sealed record ExtensionResponse(string? SecretString);
}

The extension's cache TTL is set with the environment variable SECRETS_MANAGER_TTL (seconds) ⚠ VERIFY the default and maximum in the extension documentation.

Code explained line by line

  1. AddSystemsManager("/hl-m05/accounts-api/", optional: true, reloadAfter: ...) — reads every parameter under the path with GetParametersByPath, strips the prefix and turns / into :, so /hl-m05/accounts-api/Features/Reveal becomes Features:Reveal. optional: true lets the app start if the path is empty; in production prefer false so a missing configuration fails fast.
  2. reloadAfter — background reload. Useful for log levels and feature flags; do not rely on it for secrets.
  3. AddDefaultAWSOptions(builder.Configuration.GetAWSOptions()) + AddAWSService<IAmazonSecretsManager>() — the SDK v4 way to register clients; profile and Region come from the AWS section or environment.
  4. SecretsManagerCache — AWS's caching client (version 3.x targets SDK v4). CacheItemTTL is in milliseconds; one hour is its default. A singleton cache means one GetSecretValue per secret per TTL per process.
  5. The /config endpoint returns only whether a secret-like key exists. Never echo secret values in APIs, logs, exceptions or health checks.
  6. JsonSerializerDefaults.Web maps the secret's lower-case username/password to the record's properties.
  7. DbCredentials.ToString() hides the password, so an accidental log.LogInformation("{Creds}", creds) does not leak it.
  8. RefreshNowAsync forces a reload after an authentication failure. Call it once and retry once — never in a loop, or a genuinely wrong password becomes a self-inflicted throttling incident.
  9. The Lambda client calls the extension on localhost:2773 with the X-Aws-Parameters-Secrets-Token header set to the function's session token; the extension rejects requests without it.

Security considerations

⚠ Learning shortcut — not for production

The lab secret uses the AWS managed key aws/secretsmanager, has no rotation and is readable by your administrator SSO role. In production: a customer managed key with a key policy scoped by kms:ViaService, rotation enabled (managed or Lambda), a resource policy that denies everyone except the application, rotation function and break-glass role, and interface VPC endpoints so secret traffic never leaves the VPC.

  • Prefer no secret at all. RDS IAM authentication, IAM roles for AWS APIs and mTLS with short-lived certificates remove whole classes of passwords.
  • One secret per application per environment. Shared credentials cannot be rotated safely and destroy audit attribution.
  • ECS secrets in task definitions inject values at task start. After rotation, running tasks keep the old value until restarted — fine with alternating users, risky with single user. Reading through the SDK with a cache avoids this.
  • Lambda environment variables are not secret storage. Anyone with lambda:GetFunctionConfiguration can read them. Use the extension or the SDK.
  • Do not let Terraform or CloudFormation hold secret values. State files and templates are widely readable. Generate secrets inside Secrets Manager or let the owning service manage them.
  • Watch deletions. Restrict DeleteSecret with ForceDeleteWithoutRecovery and alert on DeleteSecret and PutResourcePolicy events.
  • Logging: log secret IDs and version IDs, never values. Add a test that serialises credential objects and asserts the password is absent (the lab does this).

Banking use case

Harbourline's Accounts API in production:

  • Aurora PostgreSQL cluster created with ManageMasterUserPassword; the master secret is rotated by RDS every 7 days and is readable only by the DBA break-glass role.
  • Application user accounts_api and its clone accounts_api_clone are managed by the alternating-users rotation template every 30 days, running in the database VPC.
  • The API reads hl-prod/accounts-api/db through a cache with a 15-minute TTL; on PostgreSQL error 28P01 (invalid password) it refreshes once and retries.
  • The blind-index HMAC key for IBAN search (see the module project) lives in hl-prod/accounts-api/blind-index-key. It is not rotated automatically, because rotating it requires rebuilding the index; a runbook covers manual rotation.
  • Fifty non-secret settings live under /hl-prod/accounts-api/ in Parameter Store at no charge.
  • A monthly report from CloudTrail lists every principal that called GetSecretValue on production secrets; anything other than application roles and the rotation function is investigated.

Cost: 3 secrets × USD 0.40 = USD 1.20 per month, plus API calls (about 4 tasks × 4 refreshes per hour × 730 hours ≈ 11,700 calls ≈ USD 0.06) — trivial compared with the risk removed.

Common mistakes

  • Calling GetSecretValue on every request. → Throttling and cost. Cache in-process with a TTL.
  • Caching forever. → Rotation breaks the app. Use a TTL and refresh on authentication failure.
  • Granting secretsmanager:GetSecretValue on *. → Any compromised workload can read every secret. Scope to ARNs.
  • Forgetting kms:Decrypt with a customer managed key. → AccessDeniedException mentioning KMS. Grant it with kms:ViaService.
  • Single-user rotation on a busy service without retry logic. → A burst of login failures after every rotation. Use alternating users or add refresh-and-retry.
  • Rotation Lambda without network access. → It must reach both the database and the Secrets Manager endpoint (VPC endpoint or NAT). Rotation then fails silently until someone checks.
  • Putting thousands of settings in Secrets Manager. → USD 0.40 each; use Parameter Store for configuration.

Troubleshooting

SymptomLikely causeFix
AccessDeniedException on GetSecretValueMissing IAM permission, resource-policy deny, or wildcard ARN without the random suffixUse the full ARN or name-??????; check the resource policy
AccessDeniedException ... kms:DecryptCustomer managed key policy or IAM lacks kms:DecryptGrant kms:Decrypt on the key with kms:ViaService
ResourceNotFoundExceptionWrong Region or name; secret scheduled for deletionCheck Region; restore-secret if within the recovery window
Rotation stuck with AWSPENDINGRotation function failed at set_secret or test_secretRead the function's CloudWatch logs; fix network or permissions; re-run rotate-secret
App fails for minutes after each rotationSingle-user strategy + long cache TTL, or ECS env-var injectionAlternating users, refresh-on-auth-failure, or restart tasks after rotation
ThrottlingException from SSMDefault Parameter Store throughput exceeded (often many containers starting at once)Cache config, use GetParametersByPath once at start-up, or enable higher throughput
ParameterNotFound with /aws/reference/secretsmanager/...Reference names are case-sensitive and need --with-decryptionUse the exact secret name and WithDecryption = true

Cost and cleanup

Cost of this lesson: one secret for an hour costs about USD 0.00056 (USD 0.40 per month prorated hourly); a few dozen API calls cost far less than a cent. Standard parameters are free. If you created a customer managed KMS key, add USD 1 per month prorated (cents for a day).

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Lab only: delete immediately instead of waiting for the recovery window.
aws secretsmanager delete-secret --secret-id hl-m05/accounts-api/db --force-delete-without-recovery
aws ssm delete-parameters --names /hl-m05/accounts-api/LogLevel /hl-m05/accounts-api/PartnerApiKey
# CDK / Terraform versions: cdk destroy  |  terraform destroy
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m05-secrets
aws secretsmanager list-secrets --filters Key=name,Values=hl-m05 --query 'SecretList[].Name'

Exercises

  • 🟢 Guided: Run the CLI section, then call list-secret-version-ids after the put-secret-value and explain which version has AWSCURRENT and AWSPREVIOUS. Find both GetSecretValue calls in CloudTrail.
  • 🟡 Challenge: Write a resource policy for hl-m05/accounts-api/db that denies GetSecretValue to every principal except one role ARN and any request that does not come through your VPC endpoint (aws:SourceVpce), while still allowing the rotation function. Test the policy logic offline with hl-policy evaluate by writing two scenario files.
  • 🔴 Live problem: After enabling 30-day single-user rotation on the payments database, the on-call engineer is paged every month at 02:14 with a burst of password authentication failed errors lasting about 15 minutes. Diagnose and propose two fixes, with trade-offs.

Assignment

Add a secrets folder to your portfolio repository with: a Terraform or CDK module that creates a secret with a customer managed key, a deny-by-default resource policy and optional rotation; a .NET ISecretReader implementation with caching and refresh-on-failure, and unit tests that use a fake Secrets Manager client (see labs/m05-secure-banking-api/tests/Harbourline.SecureBanking.Tests/SecretsTests.cs). Document which values go to Secrets Manager and which to Parameter Store.

Solutions

Knowledge check

Interview questions

  • "Secrets Manager or Parameter Store?" — Credentials and anything needing rotation, cross-account or cross-Region replication: Secrets Manager. Plain configuration and low-risk static values: Parameter Store Standard, which is free. Mention cost per secret and the rotation strategies.
  • "How do you rotate a database password without downtime?" — Alternating-users rotation, applications reading through a cache with a TTL, and refresh-and-retry on authentication failure. Or avoid passwords with IAM database authentication.
  • "How would you find out who read a production secret last month?" — CloudTrail GetSecretValue events for that secret ID, grouped by userIdentity.arn; for long-term analysis, query the organisation trail in the log-archive account (Athena or CloudWatch Logs Insights).
  • "What is wrong with injecting secrets as ECS environment variables?" — Values are fixed at task start, visible to anyone who can exec into or describe the container, and stale after rotation until tasks restart.

Certification practice

Completion checklist

  • I can explain AWSCURRENT, AWSPENDING and AWSPREVIOUS
  • I created a secret and a SecureString, read both, and found the reads in CloudTrail
  • I can choose single-user vs alternating-users rotation and explain why
  • I can read secrets from .NET with a cache and refresh on failure
  • I know the price of a secret, an API call and a Standard parameter
  • I deleted my lab secret and parameters

Key takeaways

  • Credentials go to Secrets Manager (USD 0.40 per secret per month); configuration goes to Parameter Store Standard (free).
  • Rotation works only if applications cache with a TTL and refresh on authentication failure; alternating users avoids failures entirely.
  • Cross-account secrets need a resource policy and a customer managed KMS key.
  • Never put secret values in code, images, environment variables, Terraform state or logs.
  • Every secret read is in CloudTrail: use it as your access report.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page