Skip to Content
Cloud Engineering Academy
3. .NET Application Development on AWS

Secrets in .NET on AWS

Store credentials and keys in AWS Secrets Manager, read them from .NET with client-side caching, survive rotation without outages, write a C# rotation function, and know why environment variables are the wrong place for secrets.

intermediate⏱ 1 h 50 min🟡 Low costDVA-C03SCS-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager 4.1.1.2) · AWSSDK.SecretsManager.Caching 3.0.0 · Lambda dotnet10 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Classify what counts as a secret, and explain why code, appsettings.json, environment variables and Terraform state are poor homes for one.
  • Compare Secrets Manager with Parameter Store SecureString, and choose correctly on rotation, auditing, cross-account access and cost.
  • Read secrets from .NET with the official client-side cache, and make clients rotation-safe with a refresh-and-retry pattern.
  • Choose a delivery pattern per host: runtime fetch, ECS secrets injection, Beanstalk environment secrets, or the Parameters and Secrets Lambda Extension.
  • Explain the four rotation steps and staging labels (AWSPENDING, AWSCURRENT, AWSPREVIOUS), and write a rotation function in C#.
  • Lock a secret down with KMS, a resource policy and least-privilege IAM.

Prerequisites

Why this matters in banking

Secrets are the keys to customer data and money movements: database passwords, the API key for the card processor, the HMAC key that pseudonymises customer IDs in logs, the client secret for the Open Banking directory. In incident reports across the financial sector, three causes recur:

  1. A secret in source control. A developer commits appsettings.Production.json "temporarily". It is in Git history forever, cloned onto every laptop and CI runner.
  2. A secret in an environment variable that leaks. A crash dump, a debug endpoint that prints Environment.GetEnvironmentVariables(), a support engineer running describe-task-definition — the value is out.
  3. A secret that never rotates. The card-processor key from 2019 is still valid. Nobody knows who has seen it. Rotating it now is a high-risk weekend change because nobody knows which systems use it.

PCI DSS expects cryptographic keys and authentication credentials to be protected and changed on a defined schedule; UK and EU data-protection law expects appropriate technical measures for personal data. Neither tells you how. Secrets Manager with rotation, KMS and CloudTrail gives you a defensible "how" — although your compliance and security teams, not this lesson, decide whether a given design meets a specific requirement.

The idea in plain language

A secret is any value that grants access or proves identity: passwords, API keys, tokens, private keys, signing keys, connection strings that contain credentials. Rule of thumb: if someone who sees it can act as your application, it is a secret.

AWS Secrets Manager stores secrets encrypted with AWS KMS, controls access with IAM and resource policies, logs every access in CloudTrail, replicates secrets to other Regions and — its key feature — rotates them automatically.

Each secret has versions. A version is labelled with staging labels:

LabelMeaning
AWSCURRENTThe value clients should use now (the default when you ask for a secret)
AWSPENDINGA new value being prepared during rotation
AWSPREVIOUSThe value that was current before the last rotation — useful as a fallback

Analogy: Secrets Manager is the bank's key safe with a duty officer. Staff sign out keys (every access logged), only staff on the list can open the safe (IAM and resource policy), the safe itself is locked with a master key held separately (KMS). And every month the officer changes the locks: they cut a new key (pending), fit the new lock (set), test that the new key opens it (test), then hand out the new key and keep the old one in an envelope marked "previous" for a little while (finish).

What problem Secrets Manager solves

Secrets ManagerParameter Store SecureString
PurposeCredentials and keys with a lifecycleConfiguration, including some sensitive values
Automatic rotationYes — managed rotation for supported services, or your own Lambda functionNo
Versions and staging labelsYes (AWSCURRENT/AWSPENDING/AWSPREVIOUS plus custom labels)Versions and labels, no rotation workflow
Resource-based policyYes (fine-grained, cross-account)No (sharing only for advanced parameters via AWS RAM)
Cross-Region replicationYesNo
Value sizeUp to 64 KB ⚠ VERIFY4 KB standard, 8 KB advanced
Random password generationGetRandomPassword APINo
Price (checked 2026-11-18)USD 0.40 per secret per month + USD 0.05 per 10,000 API callsStandard tier free; advanced USD 0.05 per parameter per month
CloudFormation dynamic reference{{resolve:secretsmanager:...}}{{resolve:ssm-secure:...}}

Choose Secrets Manager for anything that must rotate, is shared across accounts, guards customer data or money movements, or that auditors will ask about. SecureString is acceptable for low-risk sensitive configuration that does not rotate (the lab's pseudonymisation key uses it to stay free, and that is labelled as a shortcut). New AWS accounts receive Free Tier credits that cover Secrets Manager costs during the free plan period.

Managed rotation. For some services Secrets Manager rotates without any Lambda function — for example, the RDS/Aurora master user password when you create the database with --manage-master-user-password. The secret is owned by RDS, which rotates it on a schedule.

Limitations and sharp edges:

  • Every secret costs USD 0.40 per month, including replicas and secrets scheduled for deletion until they are gone ⚠ VERIFY replica pricing on the pricing page.
  • Deletion has a recovery window of 7–30 days (default 30) unless you force immediate deletion; you cannot reuse the name until it is gone.
  • Rotation changes the credential while your application is running. If your app caches the old value and the old credential is revoked, requests fail — your code must handle this (see the rotation-safe pattern below).
  • Rotation schedules can run as often as every four hours, and run within a rotation window (UTC).
  • API calls are billed and rate-limited, so never call GetSecretValue per request: cache.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Key Vault secretsAWS Secrets ManagerKeys (cryptographic) are a separate service, AWS KMS
Key Vault keys / HSMAWS KMS (and CloudHSM)KMS keys never leave KMS; Secrets Manager uses them for envelope encryption
Key Vault access policies / RBACIAM identity policies + secret resource policies + KMS key policiesThree layers can allow or deny
Key Vault references in App ServiceECS secrets, Beanstalk environmentsecrets, Lambda extensionAll inject at start or cache with TTL; rotation needs a refresh strategy
Azure.Extensions.AspNetCore.Configuration.SecretsRuntime fetch with AWSSDK.SecretsManager.CachingThere is no first-party IConfiguration provider for Secrets Manager in the AWS .NET packages; a cache service is the idiomatic approach ⚠ VERIFY
Key Vault secret rotation with Event Grid + FunctionSecrets Manager rotation with a Lambda functionBuilt-in four-step protocol and staging labels
Soft delete + purge protectionRecovery window (7–30 days)Forced immediate deletion is possible unless an SCP denies it
Managed identityTask/function/instance roleNo secret needed to fetch secrets

Architecture

Security-boundary view — three locks on one secret:

An access succeeds only if the identity policy, the secret's resource policy and the KMS key policy all allow it (and no SCP or permission boundary denies it).

Delivery patterns compared

PatternHowPicks up rotation?Secret visible as env var?Best for
Runtime fetch with cache (recommended)App calls Secrets Manager through SecretsManagerCache; TTL e.g. 15 minYes, within TTL; immediately with refresh-on-auth-failureNoAll hosts, especially rotated credentials
ECS secrets injectionExecution role fetches at task start, sets an env varNo — new tasks only (force a new deployment)YesValues that rarely change; third-party images you cannot modify
Beanstalk environmentsecretsFetched at instance bootstrap into env varsNo — until instances are replaced or environment updatedYesSimple Beanstalk apps
Lambda Parameters and Secrets ExtensionLayer serving localhost:2773/secretsmanager/get?secretId=… with a cache (TTL up to 300 s, default 300)Yes, within TTLNoLambda functions, any language
CloudFormation dynamic references{{resolve:secretsmanager:…}} in templatesNoDepends on the propertyPassing a secret to a resource at creation (e.g. a database)

Why environment variables are a weak home for secrets: the ECS documentation itself notes that applications in the container, container logs and debugging tools can see environment variables. They also appear in process listings on EC2 hosts, in crash dumps, in describe API output for some hosts, and are often printed by "dump config" debug endpoints. Injected env vars are still far better than values in source code — but runtime fetching is better again.

Step-by-step implementation

Create a customer managed KMS key (production) or use the AWS managed key (lab)

For production, a customer managed key lets the key policy restrict decryption to Secrets Manager in your account and to named roles. The lab uses the default aws/secretsmanager key.

Create the secret as JSON

Store structured values ({"keyId":"…","apiKey":"…"}) so related parts rotate together. Tag it. Generate values with get-random-password rather than typing them.

Attach a resource policy

Deny access from outside your organisation, and block public policies with --block-public-policy.

Grant the application role read access to that one secret

secretsmanager:GetSecretValue on the secret ARN (use a trailing -?????? wildcard, because Secrets Manager appends six random characters to ARNs), plus kms:Decrypt if you use a customer managed key.

Read it from .NET with the cache and rotation-safe retry

Register a singleton SecretsManagerCache and a typed accessor; use a DelegatingHandler for outbound HTTP clients that refreshes the secret once on 401 and retries.

Turn on rotation

Deploy a rotation function (or use managed rotation for RDS), allow Secrets Manager to invoke it, and set a schedule such as rate(30 days). Trigger an immediate rotation and confirm the app keeps working.

AWS Console instructions

  1. Console → Secrets Manager → Store a new secret. Type Other type of secret; key/value pairs keyId and apiKey. Encryption key aws/secretsmanager (lab) or your customer managed key.
  2. Secret name hl-m03/partner-api; description; tags Project=cea-lab, Lab=m03-secrets, Environment=learn, Owner=you. Optional Resource permissions → Edit permissions to paste a resource policy.
  3. Configure rotation – optional: leave off for now. Store.
  4. Open the secret → Overview shows the ARN (note the six-character suffix) → Retrieve secret value (this access is logged in CloudTrail).
  5. Later: Rotation configuration → Edit rotation → Automatic rotation on, schedule Schedule expression builder: 30 days or rate(4 hours) for the test, choose your rotation function, Save. Rotate secret immediately to test.
  6. Versions tab shows version IDs with AWSCURRENT / AWSPREVIOUS labels after rotation.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
NAME=hl-m03/partner-api

# 1. Generate a strong value instead of typing one (never echo real secrets into shell history in production).
KEY=$(aws secretsmanager get-random-password --password-length 48 --exclude-punctuation \
  --query RandomPassword --output text)

# 2. Create the secret as JSON. Tags satisfy CONVENTIONS.md; KMS key defaults to aws/secretsmanager.
SECRET_ARN=$(aws secretsmanager create-secret --name "$NAME" \
  --description "Fictional partner API key for the m03 secrets lesson" \
  --secret-string "{\"keyId\":\"k-001\",\"apiKey\":\"$KEY\"}" \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m03-secrets Key=Environment,Value=learn Key=Owner,Value=you \
  --query ARN --output text)
unset KEY

# 3. Resource policy: deny any principal outside this account; block public policies.
cat > secret-policy.json <<EOF
{ "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenyOutsideAccount", "Effect": "Deny", "Principal": "*",
    "Action": "secretsmanager:*", "Resource": "*",
    "Condition": { "StringNotEquals": { "aws:PrincipalAccount": "$ACCOUNT" } } }] }
EOF
aws secretsmanager put-resource-policy --secret-id "$NAME" \
  --resource-policy file://secret-policy.json --block-public-policy
aws secretsmanager validate-resource-policy --secret-id "$NAME" --resource-policy file://secret-policy.json

# 4. Identity policy for the application role (attach to your task/function/instance role).
cat > read-secret.json <<EOF
{ "Version": "2012-10-17",
  "Statement": [{ "Effect": "Allow", "Action": "secretsmanager:GetSecretValue",
    "Resource": "arn:aws:secretsmanager:us-east-1:$ACCOUNT:secret:hl-m03/partner-api-??????" }] }
EOF

# 5. Read it (logged in CloudTrail) and look at versions and labels.
aws secretsmanager get-secret-value --secret-id "$NAME" --query '{v:VersionId,stages:VersionStages}'
aws secretsmanager list-secret-version-ids --secret-id "$NAME" \
  --query 'Versions[].{id:VersionId,stages:VersionStages}' --output table

# 6. Rotation with your own Lambda function (code below). Allow Secrets Manager to invoke it,
#    scoped to this secret, then set the schedule and rotate immediately.
aws lambda add-permission --function-name hl-m03-rotate-partner-key --statement-id secretsmanager \
  --action lambda:InvokeFunction --principal secretsmanager.amazonaws.com \
  --source-account "$ACCOUNT" --source-arn "$SECRET_ARN"
aws secretsmanager rotate-secret --secret-id "$NAME" \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:$ACCOUNT:function:hl-m03-rotate-partner-key \
  --rotation-rules '{"ScheduleExpression":"rate(30 days)","Duration":"2h"}'
# (rotate-secret also starts an immediate rotation; watch the function's logs.)

# 7. Who read the secret? CloudTrail management events (last 90 days are in Event history).
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue="$SECRET_ARN" \
  --max-results 10 --query 'Events[].{time:EventTime,name:EventName,user:Username}' --output table

Infrastructure as Code

SecretsStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.SecretsManager;
using Constructs;

public sealed class SecretsStack : Stack
{
    public SecretsStack(Construct scope, string id, IStackProps props, IRole appRole) : base(scope, id, props)
    {
        // Customer managed key: rotation of the key itself on, and usable only through Secrets Manager.
        var key = new Key(this, "SecretsKey", new KeyProps
        {
            Alias = "alias/hl-m03-secrets",
            EnableKeyRotation = true,
            RemovalPolicy = RemovalPolicy.DESTROY, // lab only; production keys are retained
        });

        // CDK generates the value at deploy time: it never appears in source code or the template.
        var secret = new Secret(this, "PartnerApi", new SecretProps
        {
            SecretName = "hl-m03/partner-api",
            EncryptionKey = key,
            GenerateSecretString = new SecretStringGenerator
            {
                SecretStringTemplate = """{"keyId":"k-001"}""",
                GenerateStringKey = "apiKey",
                PasswordLength = 48,
                ExcludePunctuation = true,
            },
        });

        // Grants GetSecretValue/DescribeSecret on THIS secret and Decrypt on its key (via Secrets Manager only).
        secret.GrantRead(appRole);

        // Deny anyone outside the account, even if another policy allowed it.
        secret.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
        {
            Effect = Effect.DENY,
            Principals = [new AnyPrincipal()],
            Actions = ["secretsmanager:*"],
            Resources = ["*"],
            Conditions = new Dictionary<string, object>
            {
                ["StringNotEquals"] = new Dictionary<string, string> { ["aws:PrincipalAccount"] = Account },
            },
        }));
    }
}

For an RDS database, prefer managed rotation: DatabaseInstance with Credentials = Credentials.FromGeneratedSecret("hl_admin") plus AddRotationSingleUser(), or the RDS-managed master user secret.

Complete code

1. Reading secrets with a cache and refresh-on-401

Harbourline.Secrets.Demo.csproj
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.Secrets.Demo</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.SecretsManager" Version="4.1.1.2" />
    <PackageReference Include="AWSSDK.SecretsManager.Caching" Version="3.0.0" />
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
  </ItemGroup>
</Project>
PartnerApiSecret.cs
using System.Text.Json;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Extensions.Caching;
using Microsoft.Extensions.Options;

namespace Harbourline.Secrets.Demo;

public sealed class PartnerApiOptions
{
    public const string SectionName = "PartnerApi";
    public string SecretId { get; set; } = "";             // name or ARN — not the secret itself
    public Uri BaseAddress { get; set; } = new("https://partner.example.net/");
    public TimeSpan SecretCacheTtl { get; set; } = TimeSpan.FromMinutes(15);
}

/// <summary>The JSON stored in Secrets Manager. Never log an instance of this type.</summary>
public sealed record PartnerApiCredential(string KeyId, string ApiKey)
{
    public override string ToString() => $"PartnerApiCredential {{ KeyId = {KeyId}, ApiKey = *** }}";
}

/// <summary>
/// Singleton accessor. The cache makes at most one Secrets Manager call per TTL per process,
/// and RefreshAsync forces a fresh read (used after an authentication failure during rotation).
/// </summary>
public sealed class PartnerApiSecret : IDisposable
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
    private readonly SecretsManagerCache _cache;
    private readonly string _secretId;

    public PartnerApiSecret(IAmazonSecretsManager client, IOptions<PartnerApiOptions> options)
    {
        _secretId = string.IsNullOrWhiteSpace(options.Value.SecretId)
            ? throw new InvalidOperationException("PartnerApi:SecretId is not configured.")
            : options.Value.SecretId;
        _cache = new SecretsManagerCache(client, new SecretCacheConfiguration
        {
            CacheItemTTL = (uint)options.Value.SecretCacheTtl.TotalMilliseconds,
        });
    }

    public async Task<PartnerApiCredential> GetAsync(CancellationToken cancellationToken)
    {
        var json = await _cache.GetSecretString(_secretId, cancellationToken);
        return JsonSerializer.Deserialize<PartnerApiCredential>(json, Json)
            ?? throw new InvalidOperationException("Secret is not valid PartnerApiCredential JSON.");
    }

    public Task RefreshAsync(CancellationToken cancellationToken) =>
        _cache.RefreshNowAsync(_secretId, cancellationToken);

    public void Dispose() => _cache.Dispose();
}

/// <summary>
/// Adds the API key to every outbound request. On 401, refreshes the secret once and retries,
/// which covers the window where the partner has switched keys but our cache has not expired yet.
/// </summary>
public sealed class PartnerApiKeyHandler(PartnerApiSecret secret, ILogger<PartnerApiKeyHandler> logger) : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var credential = await secret.GetAsync(cancellationToken);
        Apply(request, credential);
        var response = await base.SendAsync(request, cancellationToken);

        if (response.StatusCode != System.Net.HttpStatusCode.Unauthorized || !CanRetry(request))
        {
            return response;
        }

        logger.LogWarning("Partner API returned 401 with key {KeyId}; refreshing secret and retrying once.", credential.KeyId);
        response.Dispose();
        await secret.RefreshAsync(cancellationToken);
        var refreshed = await secret.GetAsync(cancellationToken);

        using var retry = await CloneAsync(request, cancellationToken);
        Apply(retry, refreshed);
        return await base.SendAsync(retry, cancellationToken);
    }

    private static void Apply(HttpRequestMessage request, PartnerApiCredential credential)
    {
        request.Headers.Remove("X-Api-Key");
        request.Headers.Remove("X-Api-Key-Id");
        request.Headers.Add("X-Api-Key", credential.ApiKey);
        request.Headers.Add("X-Api-Key-Id", credential.KeyId);
    }

    // Only GET/HEAD (no body) are retried here; POSTs that move money need an Idempotency-Key first.
    private static bool CanRetry(HttpRequestMessage request) =>
        request.Method == HttpMethod.Get || request.Method == HttpMethod.Head;

    private static async Task<HttpRequestMessage> CloneAsync(HttpRequestMessage original, CancellationToken cancellationToken)
    {
        var clone = new HttpRequestMessage(original.Method, original.RequestUri) { Version = original.Version };
        foreach (var header in original.Headers)
        {
            clone.Headers.TryAddWithoutValidation(header.Key, header.Value);
        }
        if (original.Content is not null)
        {
            var bytes = await original.Content.ReadAsByteArrayAsync(cancellationToken);
            clone.Content = new ByteArrayContent(bytes);
            foreach (var header in original.Content.Headers)
            {
                clone.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
            }
        }
        return clone;
    }
}
Program.cs
using Amazon.SecretsManager;
using Harbourline.Secrets.Demo;

var builder = WebApplication.CreateBuilder(args);

builder.Services.Configure<PartnerApiOptions>(builder.Configuration.GetSection(PartnerApiOptions.SectionName));
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSecretsManager>();
builder.Services.AddSingleton<PartnerApiSecret>();
builder.Services.AddTransient<PartnerApiKeyHandler>();

builder.Services.AddHttpClient("partner", (sp, client) =>
    {
        var options = sp.GetRequiredService<Microsoft.Extensions.Options.IOptions<PartnerApiOptions>>().Value;
        client.BaseAddress = options.BaseAddress;
        client.Timeout = TimeSpan.FromSeconds(5);
    })
    .AddHttpMessageHandler<PartnerApiKeyHandler>();

var app = builder.Build();

// Example: proxy a balance lookup to the partner. Returns the partner's status, never the key.
app.MapGet("/partner/status", async (IHttpClientFactory factory, CancellationToken ct) =>
{
    using var response = await factory.CreateClient("partner").GetAsync("status", ct);
    return Results.StatusCode((int)response.StatusCode);
});

// Prove the secret can be read without printing it.
app.MapGet("/partner/key-id", async (PartnerApiSecret secret, CancellationToken ct) =>
    Results.Ok(new { (await secret.GetAsync(ct)).KeyId }));

app.Run();
appsettings.json
{
  "AWS": { "Region": "us-east-1" },
  "PartnerApi": {
    "SecretId": "hl-m03/partner-api",
    "BaseAddress": "https://partner.example.net/",
    "SecretCacheTtl": "00:15:00"
  }
}

2. A rotation function in C#

Secrets Manager calls the rotation function four times per rotation, once per step, with the same ClientRequestToken (the new version's ID). This function rotates the fictional partner key through an IPartnerKeyAdmin interface — in real life, the partner's key-management API.

RotatePartnerKey/Function.cs
using Amazon.Lambda.Core;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;
using System.Text.Json;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Secrets.Rotation;

public sealed record RotationEvent(string Step, string SecretId, string ClientRequestToken);

/// <summary>Abstraction over the partner's key-management API (fictional).</summary>
public interface IPartnerKeyAdmin
{
    Task RegisterKeyAsync(string keyId, string apiKey, CancellationToken ct);   // new key valid alongside old
    Task<bool> TestKeyAsync(string keyId, string apiKey, CancellationToken ct);
    Task RevokeKeyAsync(string keyId, CancellationToken ct);
}

public sealed class Function(IAmazonSecretsManager secrets, IPartnerKeyAdmin partner)
{
    private const string Current = "AWSCURRENT", Pending = "AWSPENDING";
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);

    // Lambda needs a parameterless constructor; tests use the other one with fakes.
    public Function() : this(new AmazonSecretsManagerClient(), new NotConfiguredPartnerAdmin()) { }

    public async Task Handler(RotationEvent e, ILambdaContext context)
    {
        using var cts = new CancellationTokenSource(context.RemainingTime - TimeSpan.FromSeconds(2));
        var ct = cts.Token;

        // Guard rails: rotation must be enabled and the token must be a known, not-yet-current version.
        var meta = await secrets.DescribeSecretAsync(new DescribeSecretRequest { SecretId = e.SecretId }, ct);
        if (meta.RotationEnabled != true) throw new InvalidOperationException("Rotation is not enabled.");
        var versions = meta.VersionIdsToStages ?? new Dictionary<string, List<string>>();
        if (!versions.TryGetValue(e.ClientRequestToken, out var stages))
        {
            throw new InvalidOperationException("Unknown version token.");
        }
        if (stages.Contains(Current))
        {
            context.Logger.LogInformation("Version already current; nothing to do.");
            return;
        }
        if (!stages.Contains(Pending))
        {
            throw new InvalidOperationException("Version is not AWSPENDING.");
        }

        context.Logger.LogInformation($"Rotation step {e.Step} for {meta.Name}");
        switch (e.Step)
        {
            case "createSecret":
                await CreateAsync(e, ct);
                break;
            case "setSecret":
            {
                var pending = await PendingAsync(e, ct);
                await partner.RegisterKeyAsync(pending.KeyId, pending.ApiKey, ct);
                break;
            }
            case "testSecret":
            {
                var pending = await PendingAsync(e, ct);
                if (!await partner.TestKeyAsync(pending.KeyId, pending.ApiKey, ct))
                {
                    throw new InvalidOperationException("New key failed the partner test call.");
                }
                break;
            }
            case "finishSecret":
                await FinishAsync(e, versions, ct);
                break;
            default:
                throw new ArgumentException($"Unknown step {e.Step}");
        }
    }

    // Step 1: create the new value as AWSPENDING (idempotent: do nothing if it already exists).
    private async Task CreateAsync(RotationEvent e, CancellationToken ct)
    {
        try
        {
            await secrets.GetSecretValueAsync(new GetSecretValueRequest
            {
                SecretId = e.SecretId, VersionId = e.ClientRequestToken, VersionStage = Pending,
            }, ct);
            return; // a retry of this step: the pending value is already there
        }
        catch (ResourceNotFoundException) { }

        var current = await ReadAsync(e.SecretId, Current, ct);
        var random = await secrets.GetRandomPasswordAsync(new GetRandomPasswordRequest
        {
            PasswordLength = 48, ExcludePunctuation = true,
        }, ct);
        var next = current with { KeyId = NextKeyId(current.KeyId), ApiKey = random.RandomPassword };

        await secrets.PutSecretValueAsync(new PutSecretValueRequest
        {
            SecretId = e.SecretId,
            ClientRequestToken = e.ClientRequestToken,
            SecretString = JsonSerializer.Serialize(next, Json),
            VersionStages = [Pending],
        }, ct);
    }

    // Step 4: move AWSCURRENT to the new version (the old one automatically becomes AWSPREVIOUS),
    // then revoke the old key at the partner.
    private async Task FinishAsync(RotationEvent e, Dictionary<string, List<string>> versions, CancellationToken ct)
    {
        var currentVersionId = versions.First(v => v.Value.Contains(Current)).Key;
        var old = await ReadAsync(e.SecretId, Current, ct);

        await secrets.UpdateSecretVersionStageAsync(new UpdateSecretVersionStageRequest
        {
            SecretId = e.SecretId,
            VersionStage = Current,
            MoveToVersionId = e.ClientRequestToken,
            RemoveFromVersionId = currentVersionId,
        }, ct);

        // Clients cache for up to their TTL. Revoking immediately would break them; a real design
        // revokes in a later scheduled job after max(cache TTL) has passed. Shown here for completeness.
        await partner.RevokeKeyAsync(old.KeyId, ct);
    }

    private Task<Credential> PendingAsync(RotationEvent e, CancellationToken ct) =>
        ReadAsync(e.SecretId, Pending, ct, e.ClientRequestToken);

    private async Task<Credential> ReadAsync(string secretId, string stage, CancellationToken ct, string? versionId = null)
    {
        var value = await secrets.GetSecretValueAsync(new GetSecretValueRequest
        {
            SecretId = secretId, VersionStage = stage, VersionId = versionId,
        }, ct);
        return JsonSerializer.Deserialize<Credential>(value.SecretString, Json)
            ?? throw new InvalidOperationException("Secret is not valid JSON.");
    }

    private static string NextKeyId(string keyId) =>
        int.TryParse(keyId.AsSpan(2), out var n) ? $"k-{n + 1:000}" : "k-001";

    public sealed record Credential(string KeyId, string ApiKey);

    private sealed class NotConfiguredPartnerAdmin : IPartnerKeyAdmin
    {
        public Task RegisterKeyAsync(string keyId, string apiKey, CancellationToken ct) => throw new NotSupportedException("Wire the partner client here.");
        public Task<bool> TestKeyAsync(string keyId, string apiKey, CancellationToken ct) => throw new NotSupportedException();
        public Task RevokeKeyAsync(string keyId, CancellationToken ct) => throw new NotSupportedException();
    }
}

The rotation function's execution role needs secretsmanager:DescribeSecret, GetSecretValue, PutSecretValue and UpdateSecretVersionStage on this secret, secretsmanager:GetRandomPassword (resource *), kms:Decrypt and kms:GenerateDataKey on the key if it is customer managed, and network access to the partner API and to Secrets Manager.

Code explained line by line

Reading

  1. PartnerApiOptions.SecretId — configuration holds the secret's name, not its value. Names are not sensitive.
  2. PartnerApiCredential.ToString() override — if someone logs the object, the key is masked. Defence in depth against accidental logging.
  3. new SecretsManagerCache(client, config) with CacheItemTTL — the official cache from AWSSDK.SecretsManager.Caching. One instance per process (singleton), so the whole app shares it.
  4. GetSecretString(secretId, ct) — returns the cached AWSCURRENT string, calling Secrets Manager only when the TTL has expired. This cuts both cost (USD 0.05 per 10,000 calls) and latency.
  5. RefreshNowAsync — forces an immediate refresh of one secret. Used only after an authentication failure, never per request.
  6. PartnerApiKeyHandler — an HttpClient DelegatingHandler, the same pattern as an auth handler in Azure SDK clients. It applies the key and, on 401, refreshes once and retries once.
  7. CanRetry — only safe, body-less methods are retried automatically. A payment POST must never be blindly replayed; it needs an Idempotency-Key header so the partner can de-duplicate.
  8. /partner/key-id endpoint — returns the key ID to prove the wiring works without exposing the key. Remove it in production.

Rotation

  1. DescribeSecretAsync + VersionIdsToStages — the guard rails every rotation function needs: rotation enabled, the token is a known version, and it is AWSPENDING (or already AWSCURRENT, in which case a retry is a no-op).
  2. createSecret — idempotent: if the pending version already exists (Secrets Manager retries steps), do nothing. Otherwise GetRandomPassword + PutSecretValue with VersionStages = ["AWSPENDING"] and the same ClientRequestToken.
  3. setSecret — makes the new credential valid at the downstream system (the partner, or the database's ALTER USER).
  4. testSecret — proves the new credential works before clients are switched.
  5. finishSecret — UpdateSecretVersionStage moves AWSCURRENT to the new version; Secrets Manager automatically labels the old one AWSPREVIOUS.
  6. Revoking the old key — only after every client's cache has expired. The comment explains why immediate revocation is dangerous; in production, a scheduled job revokes AWSPREVIOUS after the maximum cache TTL.
  7. RemainingTime - 2 s — a timeout budget so the function fails cleanly instead of being killed mid-step.

Rotation strategies for databases

  • Single user: one database user whose password changes. Simple, but there is a short window where connections opened with the old password fail until clients refresh. Fine with refresh-on-failure logic and connection retry.
  • Alternating users: two users (app_a, app_b); rotation updates the inactive one, then switches AWSCURRENT. Clients holding the old credentials keep working until the next rotation. Requires a superuser secret for the rotation function. Preferred for high-availability systems.

Security considerations

⚠ Learning shortcut — not for production

The lab encrypts the secret with the AWS managed key aws/secretsmanager, uses an account-scoped resource policy, and the rotation function revokes the old key immediately. In production: a customer managed KMS key whose key policy limits kms:Decrypt to named roles via kms:ViaService = secretsmanager.<region>.amazonaws.com; a resource policy that also requires aws:PrincipalOrgID and, for workloads in VPCs, access through a Secrets Manager VPC endpoint (aws:SourceVpce); a delayed revocation of AWSPREVIOUS; and alarms on rotation failures (EventBridge events from Secrets Manager).

  • Least privilege: one secret ARN per statement (with the -?????? suffix wildcard), GetSecretValue only. Never secretsmanager:* on * for an application.
  • Separate duties: humans should not routinely read production secrets. Grant break-glass access through a separate role with MFA and alerting on every GetSecretValue by a human principal.
  • CloudTrail: GetSecretValue is a logged management event. Alert on reads by unexpected principals or from unexpected networks.
  • No secrets in Terraform state, CDK context, pipeline logs or container images. Generate them in AWS (CDK GenerateSecretString, get-random-password) so no human sees them.
  • SCP guard rails: deny secretsmanager:DeleteSecret with ForceDeleteWithoutRecovery and deny secretsmanager:PutResourcePolicy without BlockPublicPolicy across the organisation.
  • Memory hygiene: .NET strings are immutable and cannot be reliably wiped. Keep secrets in as few places as possible, do not copy them into exceptions, logs or metrics, and avoid long-lived static copies outside the cache.
  • Cross-Region DR: replicate critical secrets to the recovery Region so failover does not depend on the primary Region being reachable.

Banking use case

Harbourline's Payments Service connects to the card processor with a key pair and to PostgreSQL with a database user. The database secret is the RDS-managed master secret for admin tasks plus an application user rotated with the alternating users strategy every 30 days. The card-processor key rotates every 90 days through a custom C# rotation function very like the one above, which calls the processor's key-management API; the old key is revoked 24 hours later by a scheduled job. Every application reads secrets at runtime through SecretsManagerCache with a 15-minute TTL and refresh-on-401.

The security team gets three dashboards from CloudTrail and EventBridge: rotation success/failure per secret, GetSecretValue by human principals (should be zero outside incidents), and secrets with rotation disabled. The quarterly access review uses IAM Access Analyzer to list every principal that can read each secret. Compliance reviewers still own the decision on whether this meets the bank's key-management policy; the platform provides the evidence.

Common mistakes

  • Calling GetSecretValue on every request → cost, latency, throttling → cache with a TTL.
  • Caching forever (reading once at start-up into a static) → rotation breaks the app until restart → TTL plus refresh-on-auth-failure.
  • Injecting secrets as environment variables and expecting rotation to work → tasks keep the old value → runtime fetch, or force a new deployment after rotation.
  • IAM resource ARN without the random suffix → AccessDenied → use name-?????? or the full ARN.
  • Customer managed key without granting the app kms:Decrypt → AccessDeniedException mentioning KMS → add it in the key policy or IAM policy.
  • Non-idempotent rotation steps → duplicate pending versions or a half-rotated secret after a retry → check for existing AWSPENDING first.
  • Revoking the old credential in finishSecret → clients with cached values fail for up to one TTL → revoke later.
  • Logging the secret in an exception message → secret in CloudWatch Logs → mask in ToString(), never include values in messages.

Troubleshooting

SymptomLikely causeFix
AccessDeniedException ... secretsmanager:GetSecretValueIdentity policy ARN mismatch or resource policy denyCheck the -?????? suffix; run validate-resource-policy; check SCPs
AccessDeniedException ... kms:DecryptCMK key policy or IAM lacks Decrypt for the roleAdd Decrypt with kms:ViaService condition
ResourceNotFoundExceptionWrong Region or name, or secret scheduled for deletionCheck AWS_REGION; restore-secret if within the recovery window
App works, then fails after rotation until restartValue cached without TTL or never refreshedUse the cache TTL and refresh-on-401 pattern
Rotation stuck: AWSPENDING exists on an old versionA previous rotation failed midwayFix the function; remove the stale AWSPENDING label with update-secret-version-stage; rotate again
Rotation Lambda times outNo network path to Secrets Manager or the target (VPC without endpoint/NAT)Add a Secrets Manager VPC endpoint and a route to the target
ECS task: ResourceInitializationError: unable to retrieve secretExecution role lacks GetSecretValue / no network pathFix execution role (not task role); add endpoints
ThrottlingException on GetSecretValueUncached reads at high request ratesCache; increase TTL

Cost and cleanup

What this lesson costs: one secret for a day costs about USD 0.013 (USD 0.40 per month, prorated); a few hundred API calls cost well under one cent. A rotation Lambda runs within the free tier. A customer managed KMS key costs about USD 1 per month prorated ⚠ VERIFY, plus requests. Accounts created after 2025-07-15 have Free Tier credits that cover this. Label: credits (cents).

Free alternative: use a Parameter Store SecureString for the reading part (free) and do the rotation part as a design exercise with the C# function and unit tests only.

Teardown:

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws secretsmanager cancel-rotate-secret --secret-id hl-m03/partner-api 2>/dev/null || true
# Lab only: skip the recovery window so the name can be reused and billing stops now.
aws secretsmanager delete-secret --secret-id hl-m03/partner-api --force-delete-without-recovery
aws lambda delete-function --function-name hl-m03-rotate-partner-key 2>/dev/null || true
# If you created a CMK: schedule deletion (minimum 7 days); it stops being usable immediately when disabled.
# aws kms schedule-key-deletion --key-id alias/hl-m03-secrets --pending-window-in-days 7
aws secretsmanager list-secrets --filters Key=tag-key,Values=Project --query 'SecretList[].Name'

Exercises

  • 🟢 Guided: Create the secret with the CLI, grant your local SSO role read access, run the demo app, and call /partner/key-id. Then update the secret value with put-secret-value and confirm the app shows the new key ID after the cache TTL (set the TTL to 30 seconds for the test).
  • 🟡 Challenge: Write unit tests for PartnerApiKeyHandler with a fake inner handler that returns 401 for the old key and 200 for the new one, and a fake secret source. Assert exactly one refresh and one retry, and no retry for POST.
  • 🔴 Live problem: Ticket SEC-3307: "Every 30 days, at 03:00, the Payments API logs about 2 minutes of 28P01 password authentication failed errors and the on-call engineer is paged." The secret uses the single-user rotation strategy and the app reads it at start-up into a static field. Propose a fix and a rollout plan.

Assignment

Write a short secrets standard for your portfolio: classification of secret types, where each may live, delivery pattern per host, rotation period per type, required IAM/KMS/resource-policy controls and monitoring. Include the PartnerApiSecret and PartnerApiKeyHandler classes with tests as the reference implementation.

Solutions

Knowledge check

Interview questions

  • "How do you manage secrets for a .NET service on AWS?" — Secrets Manager with a CMK and resource policy; the service role can read only its secrets; runtime fetch through SecretsManagerCache with a TTL and refresh-on-auth-failure; automatic rotation (managed for RDS, Lambda otherwise); CloudTrail monitoring; no secrets in code, images, env vars or IaC state.
  • "Explain how rotation works without downtime." — Four steps with staging labels; the new credential is valid before clients switch; clients refresh within a TTL; the old one stays valid (alternating users, delayed revocation) until all caches have expired.
  • "Why not environment variables?" — Visible to anything in the process and to describe/debug tooling, captured in dumps and accidental logs, and frozen at start so rotation needs a redeploy.
  • "How would you detect secret misuse?" — CloudTrail GetSecretValue events by unexpected principals, source IPs or VPC endpoints; Access Analyzer for external access; EventBridge alerts on rotation failures and policy changes.

Certification practice

Completion checklist

  • I can explain why environment variables, config files and IaC state are poor places for secrets
  • I created a secret with a resource policy and read it through the cache
  • I can describe the four rotation steps and the three staging labels
  • My client code survives a rotation without a restart
  • I force-deleted the lab secret and checked nothing remains

Key takeaways

  • Secrets Manager for anything that rotates or guards customer data; SecureString only for low-risk, non-rotating values.
  • Read secrets at runtime through a cache with a TTL; refresh once on authentication failure.
  • Rotation is createSecret → setSecret → testSecret → finishSecret, driven by staging labels; keep the old credential valid for at least one client TTL.
  • Three locks: IAM, the secret resource policy and the KMS key policy; watch access with CloudTrail.
  • Injected secrets (ECS, Beanstalk) are frozen at start; plan redeployments or prefer runtime fetching.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page