Secrets in .NET on AWS
Store credentials and keys in AWS Secrets Manager, read them from .NET with client-side caching, survive rotation without outages, write a C# rotation function, and know why environment variables are the wrong place for secrets.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager 4.1.1.2) · AWSSDK.SecretsManager.Caching 3.0.0 · Lambda dotnet10 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Classify what counts as a secret, and explain why code,
appsettings.json, environment variables and Terraform state are poor homes for one. - Compare Secrets Manager with Parameter Store
SecureString, and choose correctly on rotation, auditing, cross-account access and cost. - Read secrets from .NET with the official client-side cache, and make clients rotation-safe with a refresh-and-retry pattern.
- Choose a delivery pattern per host: runtime fetch, ECS
secretsinjection, Beanstalk environment secrets, or the Parameters and Secrets Lambda Extension. - Explain the four rotation steps and staging labels (
AWSPENDING,AWSCURRENT,AWSPREVIOUS), and write a rotation function in C#. - Lock a secret down with KMS, a resource policy and least-privilege IAM.
Prerequisites
- Configuration done right — configuration layering and
IOptionsMonitor. - IAM in depth and the KMS material in Module 5.
- One of the hosting lessons: ECS on Fargate, Elastic Beanstalk or Lambda with C#.
- Tools: .NET 10 SDK, AWS CLI v2 with
learn-sandbox.
Why this matters in banking
Secrets are the keys to customer data and money movements: database passwords, the API key for the card processor, the HMAC key that pseudonymises customer IDs in logs, the client secret for the Open Banking directory. In incident reports across the financial sector, three causes recur:
- A secret in source control. A developer commits
appsettings.Production.json"temporarily". It is in Git history forever, cloned onto every laptop and CI runner. - A secret in an environment variable that leaks. A crash dump, a debug endpoint that prints
Environment.GetEnvironmentVariables(), a support engineer runningdescribe-task-definition— the value is out. - A secret that never rotates. The card-processor key from 2019 is still valid. Nobody knows who has seen it. Rotating it now is a high-risk weekend change because nobody knows which systems use it.
PCI DSS expects cryptographic keys and authentication credentials to be protected and changed on a defined schedule; UK and EU data-protection law expects appropriate technical measures for personal data. Neither tells you how. Secrets Manager with rotation, KMS and CloudTrail gives you a defensible "how" — although your compliance and security teams, not this lesson, decide whether a given design meets a specific requirement.
The idea in plain language
A secret is any value that grants access or proves identity: passwords, API keys, tokens, private keys, signing keys, connection strings that contain credentials. Rule of thumb: if someone who sees it can act as your application, it is a secret.
AWS Secrets Manager stores secrets encrypted with AWS KMS, controls access with IAM and resource policies, logs every access in CloudTrail, replicates secrets to other Regions and — its key feature — rotates them automatically.
Each secret has versions. A version is labelled with staging labels:
| Label | Meaning |
|---|---|
AWSCURRENT | The value clients should use now (the default when you ask for a secret) |
AWSPENDING | A new value being prepared during rotation |
AWSPREVIOUS | The value that was current before the last rotation — useful as a fallback |
Analogy: Secrets Manager is the bank's key safe with a duty officer. Staff sign out keys (every access logged), only staff on the list can open the safe (IAM and resource policy), the safe itself is locked with a master key held separately (KMS). And every month the officer changes the locks: they cut a new key (pending), fit the new lock (set), test that the new key opens it (test), then hand out the new key and keep the old one in an envelope marked "previous" for a little while (finish).
What problem Secrets Manager solves
| Secrets Manager | Parameter Store SecureString | |
|---|---|---|
| Purpose | Credentials and keys with a lifecycle | Configuration, including some sensitive values |
| Automatic rotation | Yes — managed rotation for supported services, or your own Lambda function | No |
| Versions and staging labels | Yes (AWSCURRENT/AWSPENDING/AWSPREVIOUS plus custom labels) | Versions and labels, no rotation workflow |
| Resource-based policy | Yes (fine-grained, cross-account) | No (sharing only for advanced parameters via AWS RAM) |
| Cross-Region replication | Yes | No |
| Value size | Up to 64 KB ⚠ VERIFY | 4 KB standard, 8 KB advanced |
| Random password generation | GetRandomPassword API | No |
| Price (checked 2026-11-18) | USD 0.40 per secret per month + USD 0.05 per 10,000 API calls | Standard tier free; advanced USD 0.05 per parameter per month |
| CloudFormation dynamic reference | {{resolve:secretsmanager:...}} | {{resolve:ssm-secure:...}} |
Choose Secrets Manager for anything that must rotate, is shared across accounts, guards customer data or money movements, or that auditors will ask about. SecureString is acceptable for low-risk sensitive configuration that does not rotate (the lab's pseudonymisation key uses it to stay free, and that is labelled as a shortcut). New AWS accounts receive Free Tier credits that cover Secrets Manager costs during the free plan period.
Managed rotation. For some services Secrets Manager rotates without any Lambda function — for example, the RDS/Aurora master user password when you create the database with --manage-master-user-password. The secret is owned by RDS, which rotates it on a schedule.
Limitations and sharp edges:
- Every secret costs USD 0.40 per month, including replicas and secrets scheduled for deletion until they are gone ⚠ VERIFY replica pricing on the pricing page.
- Deletion has a recovery window of 7–30 days (default 30) unless you force immediate deletion; you cannot reuse the name until it is gone.
- Rotation changes the credential while your application is running. If your app caches the old value and the old credential is revoked, requests fail — your code must handle this (see the rotation-safe pattern below).
- Rotation schedules can run as often as every four hours, and run within a rotation window (UTC).
- API calls are billed and rate-limited, so never call
GetSecretValueper request: cache.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Key Vault secrets | AWS Secrets Manager | Keys (cryptographic) are a separate service, AWS KMS |
| Key Vault keys / HSM | AWS KMS (and CloudHSM) | KMS keys never leave KMS; Secrets Manager uses them for envelope encryption |
| Key Vault access policies / RBAC | IAM identity policies + secret resource policies + KMS key policies | Three layers can allow or deny |
| Key Vault references in App Service | ECS secrets, Beanstalk environmentsecrets, Lambda extension | All inject at start or cache with TTL; rotation needs a refresh strategy |
Azure.Extensions.AspNetCore.Configuration.Secrets | Runtime fetch with AWSSDK.SecretsManager.Caching | There is no first-party IConfiguration provider for Secrets Manager in the AWS .NET packages; a cache service is the idiomatic approach ⚠ VERIFY |
| Key Vault secret rotation with Event Grid + Function | Secrets Manager rotation with a Lambda function | Built-in four-step protocol and staging labels |
| Soft delete + purge protection | Recovery window (7–30 days) | Forced immediate deletion is possible unless an SCP denies it |
| Managed identity | Task/function/instance role | No secret needed to fetch secrets |
Architecture
Security-boundary view — three locks on one secret:
An access succeeds only if the identity policy, the secret's resource policy and the KMS key policy all allow it (and no SCP or permission boundary denies it).
Delivery patterns compared
| Pattern | How | Picks up rotation? | Secret visible as env var? | Best for |
|---|---|---|---|---|
| Runtime fetch with cache (recommended) | App calls Secrets Manager through SecretsManagerCache; TTL e.g. 15 min | Yes, within TTL; immediately with refresh-on-auth-failure | No | All hosts, especially rotated credentials |
ECS secrets injection | Execution role fetches at task start, sets an env var | No — new tasks only (force a new deployment) | Yes | Values that rarely change; third-party images you cannot modify |
Beanstalk environmentsecrets | Fetched at instance bootstrap into env vars | No — until instances are replaced or environment updated | Yes | Simple Beanstalk apps |
| Lambda Parameters and Secrets Extension | Layer serving localhost:2773/secretsmanager/get?secretId=… with a cache (TTL up to 300 s, default 300) | Yes, within TTL | No | Lambda functions, any language |
| CloudFormation dynamic references | {{resolve:secretsmanager:…}} in templates | No | Depends on the property | Passing a secret to a resource at creation (e.g. a database) |
Why environment variables are a weak home for secrets: the ECS documentation itself notes that applications in the container, container logs and debugging tools can see environment variables. They also appear in process listings on EC2 hosts, in crash dumps, in describe API output for some hosts, and are often printed by "dump config" debug endpoints. Injected env vars are still far better than values in source code — but runtime fetching is better again.
Step-by-step implementation
Create a customer managed KMS key (production) or use the AWS managed key (lab)
For production, a customer managed key lets the key policy restrict decryption to Secrets Manager in your account and to named roles. The lab uses the default aws/secretsmanager key.
Create the secret as JSON
Store structured values ({"keyId":"…","apiKey":"…"}) so related parts rotate together. Tag it. Generate values with get-random-password rather than typing them.
Attach a resource policy
Deny access from outside your organisation, and block public policies with --block-public-policy.
Grant the application role read access to that one secret
secretsmanager:GetSecretValue on the secret ARN (use a trailing -?????? wildcard, because Secrets Manager appends six random characters to ARNs), plus kms:Decrypt if you use a customer managed key.
Read it from .NET with the cache and rotation-safe retry
Register a singleton SecretsManagerCache and a typed accessor; use a DelegatingHandler for outbound HTTP clients that refreshes the secret once on 401 and retries.
Turn on rotation
Deploy a rotation function (or use managed rotation for RDS), allow Secrets Manager to invoke it, and set a schedule such as rate(30 days). Trigger an immediate rotation and confirm the app keeps working.
AWS Console instructions
- Console → Secrets Manager → Store a new secret. Type Other type of secret; key/value pairs
keyIdandapiKey. Encryption key aws/secretsmanager (lab) or your customer managed key. - Secret name
hl-m03/partner-api; description; tagsProject=cea-lab,Lab=m03-secrets,Environment=learn,Owner=you. Optional Resource permissions → Edit permissions to paste a resource policy. - Configure rotation – optional: leave off for now. Store.
- Open the secret → Overview shows the ARN (note the six-character suffix) → Retrieve secret value (this access is logged in CloudTrail).
- Later: Rotation configuration → Edit rotation → Automatic rotation on, schedule Schedule expression builder: 30 days or
rate(4 hours)for the test, choose your rotation function, Save. Rotate secret immediately to test. - Versions tab shows version IDs with
AWSCURRENT/AWSPREVIOUSlabels after rotation.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
NAME=hl-m03/partner-api
# 1. Generate a strong value instead of typing one (never echo real secrets into shell history in production).
KEY=$(aws secretsmanager get-random-password --password-length 48 --exclude-punctuation \
--query RandomPassword --output text)
# 2. Create the secret as JSON. Tags satisfy CONVENTIONS.md; KMS key defaults to aws/secretsmanager.
SECRET_ARN=$(aws secretsmanager create-secret --name "$NAME" \
--description "Fictional partner API key for the m03 secrets lesson" \
--secret-string "{\"keyId\":\"k-001\",\"apiKey\":\"$KEY\"}" \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m03-secrets Key=Environment,Value=learn Key=Owner,Value=you \
--query ARN --output text)
unset KEY
# 3. Resource policy: deny any principal outside this account; block public policies.
cat > secret-policy.json <<EOF
{ "Version": "2012-10-17",
"Statement": [{
"Sid": "DenyOutsideAccount", "Effect": "Deny", "Principal": "*",
"Action": "secretsmanager:*", "Resource": "*",
"Condition": { "StringNotEquals": { "aws:PrincipalAccount": "$ACCOUNT" } } }] }
EOF
aws secretsmanager put-resource-policy --secret-id "$NAME" \
--resource-policy file://secret-policy.json --block-public-policy
aws secretsmanager validate-resource-policy --secret-id "$NAME" --resource-policy file://secret-policy.json
# 4. Identity policy for the application role (attach to your task/function/instance role).
cat > read-secret.json <<EOF
{ "Version": "2012-10-17",
"Statement": [{ "Effect": "Allow", "Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-east-1:$ACCOUNT:secret:hl-m03/partner-api-??????" }] }
EOF
# 5. Read it (logged in CloudTrail) and look at versions and labels.
aws secretsmanager get-secret-value --secret-id "$NAME" --query '{v:VersionId,stages:VersionStages}'
aws secretsmanager list-secret-version-ids --secret-id "$NAME" \
--query 'Versions[].{id:VersionId,stages:VersionStages}' --output table
# 6. Rotation with your own Lambda function (code below). Allow Secrets Manager to invoke it,
# scoped to this secret, then set the schedule and rotate immediately.
aws lambda add-permission --function-name hl-m03-rotate-partner-key --statement-id secretsmanager \
--action lambda:InvokeFunction --principal secretsmanager.amazonaws.com \
--source-account "$ACCOUNT" --source-arn "$SECRET_ARN"
aws secretsmanager rotate-secret --secret-id "$NAME" \
--rotation-lambda-arn arn:aws:lambda:us-east-1:$ACCOUNT:function:hl-m03-rotate-partner-key \
--rotation-rules '{"ScheduleExpression":"rate(30 days)","Duration":"2h"}'
# (rotate-secret also starts an immediate rotation; watch the function's logs.)
# 7. Who read the secret? CloudTrail management events (last 90 days are in Event history).
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue="$SECRET_ARN" \
--max-results 10 --query 'Events[].{time:EventTime,name:EventName,user:Username}' --output tableInfrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.KMS;
using Amazon.CDK.AWS.SecretsManager;
using Constructs;
public sealed class SecretsStack : Stack
{
public SecretsStack(Construct scope, string id, IStackProps props, IRole appRole) : base(scope, id, props)
{
// Customer managed key: rotation of the key itself on, and usable only through Secrets Manager.
var key = new Key(this, "SecretsKey", new KeyProps
{
Alias = "alias/hl-m03-secrets",
EnableKeyRotation = true,
RemovalPolicy = RemovalPolicy.DESTROY, // lab only; production keys are retained
});
// CDK generates the value at deploy time: it never appears in source code or the template.
var secret = new Secret(this, "PartnerApi", new SecretProps
{
SecretName = "hl-m03/partner-api",
EncryptionKey = key,
GenerateSecretString = new SecretStringGenerator
{
SecretStringTemplate = """{"keyId":"k-001"}""",
GenerateStringKey = "apiKey",
PasswordLength = 48,
ExcludePunctuation = true,
},
});
// Grants GetSecretValue/DescribeSecret on THIS secret and Decrypt on its key (via Secrets Manager only).
secret.GrantRead(appRole);
// Deny anyone outside the account, even if another policy allowed it.
secret.AddToResourcePolicy(new PolicyStatement(new PolicyStatementProps
{
Effect = Effect.DENY,
Principals = [new AnyPrincipal()],
Actions = ["secretsmanager:*"],
Resources = ["*"],
Conditions = new Dictionary<string, object>
{
["StringNotEquals"] = new Dictionary<string, string> { ["aws:PrincipalAccount"] = Account },
},
}));
}
}For an RDS database, prefer managed rotation: DatabaseInstance with Credentials = Credentials.FromGeneratedSecret("hl_admin") plus AddRotationSingleUser(), or the RDS-managed master user secret.
Complete code
1. Reading secrets with a cache and refresh-on-401
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Secrets.Demo</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.SecretsManager" Version="4.1.1.2" />
<PackageReference Include="AWSSDK.SecretsManager.Caching" Version="3.0.0" />
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
</ItemGroup>
</Project>using System.Text.Json;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Extensions.Caching;
using Microsoft.Extensions.Options;
namespace Harbourline.Secrets.Demo;
public sealed class PartnerApiOptions
{
public const string SectionName = "PartnerApi";
public string SecretId { get; set; } = ""; // name or ARN — not the secret itself
public Uri BaseAddress { get; set; } = new("https://partner.example.net/");
public TimeSpan SecretCacheTtl { get; set; } = TimeSpan.FromMinutes(15);
}
/// <summary>The JSON stored in Secrets Manager. Never log an instance of this type.</summary>
public sealed record PartnerApiCredential(string KeyId, string ApiKey)
{
public override string ToString() => $"PartnerApiCredential {{ KeyId = {KeyId}, ApiKey = *** }}";
}
/// <summary>
/// Singleton accessor. The cache makes at most one Secrets Manager call per TTL per process,
/// and RefreshAsync forces a fresh read (used after an authentication failure during rotation).
/// </summary>
public sealed class PartnerApiSecret : IDisposable
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
private readonly SecretsManagerCache _cache;
private readonly string _secretId;
public PartnerApiSecret(IAmazonSecretsManager client, IOptions<PartnerApiOptions> options)
{
_secretId = string.IsNullOrWhiteSpace(options.Value.SecretId)
? throw new InvalidOperationException("PartnerApi:SecretId is not configured.")
: options.Value.SecretId;
_cache = new SecretsManagerCache(client, new SecretCacheConfiguration
{
CacheItemTTL = (uint)options.Value.SecretCacheTtl.TotalMilliseconds,
});
}
public async Task<PartnerApiCredential> GetAsync(CancellationToken cancellationToken)
{
var json = await _cache.GetSecretString(_secretId, cancellationToken);
return JsonSerializer.Deserialize<PartnerApiCredential>(json, Json)
?? throw new InvalidOperationException("Secret is not valid PartnerApiCredential JSON.");
}
public Task RefreshAsync(CancellationToken cancellationToken) =>
_cache.RefreshNowAsync(_secretId, cancellationToken);
public void Dispose() => _cache.Dispose();
}
/// <summary>
/// Adds the API key to every outbound request. On 401, refreshes the secret once and retries,
/// which covers the window where the partner has switched keys but our cache has not expired yet.
/// </summary>
public sealed class PartnerApiKeyHandler(PartnerApiSecret secret, ILogger<PartnerApiKeyHandler> logger) : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var credential = await secret.GetAsync(cancellationToken);
Apply(request, credential);
var response = await base.SendAsync(request, cancellationToken);
if (response.StatusCode != System.Net.HttpStatusCode.Unauthorized || !CanRetry(request))
{
return response;
}
logger.LogWarning("Partner API returned 401 with key {KeyId}; refreshing secret and retrying once.", credential.KeyId);
response.Dispose();
await secret.RefreshAsync(cancellationToken);
var refreshed = await secret.GetAsync(cancellationToken);
using var retry = await CloneAsync(request, cancellationToken);
Apply(retry, refreshed);
return await base.SendAsync(retry, cancellationToken);
}
private static void Apply(HttpRequestMessage request, PartnerApiCredential credential)
{
request.Headers.Remove("X-Api-Key");
request.Headers.Remove("X-Api-Key-Id");
request.Headers.Add("X-Api-Key", credential.ApiKey);
request.Headers.Add("X-Api-Key-Id", credential.KeyId);
}
// Only GET/HEAD (no body) are retried here; POSTs that move money need an Idempotency-Key first.
private static bool CanRetry(HttpRequestMessage request) =>
request.Method == HttpMethod.Get || request.Method == HttpMethod.Head;
private static async Task<HttpRequestMessage> CloneAsync(HttpRequestMessage original, CancellationToken cancellationToken)
{
var clone = new HttpRequestMessage(original.Method, original.RequestUri) { Version = original.Version };
foreach (var header in original.Headers)
{
clone.Headers.TryAddWithoutValidation(header.Key, header.Value);
}
if (original.Content is not null)
{
var bytes = await original.Content.ReadAsByteArrayAsync(cancellationToken);
clone.Content = new ByteArrayContent(bytes);
foreach (var header in original.Content.Headers)
{
clone.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
}
}
return clone;
}
}using Amazon.SecretsManager;
using Harbourline.Secrets.Demo;
var builder = WebApplication.CreateBuilder(args);
builder.Services.Configure<PartnerApiOptions>(builder.Configuration.GetSection(PartnerApiOptions.SectionName));
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSecretsManager>();
builder.Services.AddSingleton<PartnerApiSecret>();
builder.Services.AddTransient<PartnerApiKeyHandler>();
builder.Services.AddHttpClient("partner", (sp, client) =>
{
var options = sp.GetRequiredService<Microsoft.Extensions.Options.IOptions<PartnerApiOptions>>().Value;
client.BaseAddress = options.BaseAddress;
client.Timeout = TimeSpan.FromSeconds(5);
})
.AddHttpMessageHandler<PartnerApiKeyHandler>();
var app = builder.Build();
// Example: proxy a balance lookup to the partner. Returns the partner's status, never the key.
app.MapGet("/partner/status", async (IHttpClientFactory factory, CancellationToken ct) =>
{
using var response = await factory.CreateClient("partner").GetAsync("status", ct);
return Results.StatusCode((int)response.StatusCode);
});
// Prove the secret can be read without printing it.
app.MapGet("/partner/key-id", async (PartnerApiSecret secret, CancellationToken ct) =>
Results.Ok(new { (await secret.GetAsync(ct)).KeyId }));
app.Run();{
"AWS": { "Region": "us-east-1" },
"PartnerApi": {
"SecretId": "hl-m03/partner-api",
"BaseAddress": "https://partner.example.net/",
"SecretCacheTtl": "00:15:00"
}
}2. A rotation function in C#
Secrets Manager calls the rotation function four times per rotation, once per step, with the same ClientRequestToken (the new version's ID). This function rotates the fictional partner key through an IPartnerKeyAdmin interface — in real life, the partner's key-management API.
using Amazon.Lambda.Core;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;
using System.Text.Json;
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace Harbourline.Secrets.Rotation;
public sealed record RotationEvent(string Step, string SecretId, string ClientRequestToken);
/// <summary>Abstraction over the partner's key-management API (fictional).</summary>
public interface IPartnerKeyAdmin
{
Task RegisterKeyAsync(string keyId, string apiKey, CancellationToken ct); // new key valid alongside old
Task<bool> TestKeyAsync(string keyId, string apiKey, CancellationToken ct);
Task RevokeKeyAsync(string keyId, CancellationToken ct);
}
public sealed class Function(IAmazonSecretsManager secrets, IPartnerKeyAdmin partner)
{
private const string Current = "AWSCURRENT", Pending = "AWSPENDING";
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
// Lambda needs a parameterless constructor; tests use the other one with fakes.
public Function() : this(new AmazonSecretsManagerClient(), new NotConfiguredPartnerAdmin()) { }
public async Task Handler(RotationEvent e, ILambdaContext context)
{
using var cts = new CancellationTokenSource(context.RemainingTime - TimeSpan.FromSeconds(2));
var ct = cts.Token;
// Guard rails: rotation must be enabled and the token must be a known, not-yet-current version.
var meta = await secrets.DescribeSecretAsync(new DescribeSecretRequest { SecretId = e.SecretId }, ct);
if (meta.RotationEnabled != true) throw new InvalidOperationException("Rotation is not enabled.");
var versions = meta.VersionIdsToStages ?? new Dictionary<string, List<string>>();
if (!versions.TryGetValue(e.ClientRequestToken, out var stages))
{
throw new InvalidOperationException("Unknown version token.");
}
if (stages.Contains(Current))
{
context.Logger.LogInformation("Version already current; nothing to do.");
return;
}
if (!stages.Contains(Pending))
{
throw new InvalidOperationException("Version is not AWSPENDING.");
}
context.Logger.LogInformation($"Rotation step {e.Step} for {meta.Name}");
switch (e.Step)
{
case "createSecret":
await CreateAsync(e, ct);
break;
case "setSecret":
{
var pending = await PendingAsync(e, ct);
await partner.RegisterKeyAsync(pending.KeyId, pending.ApiKey, ct);
break;
}
case "testSecret":
{
var pending = await PendingAsync(e, ct);
if (!await partner.TestKeyAsync(pending.KeyId, pending.ApiKey, ct))
{
throw new InvalidOperationException("New key failed the partner test call.");
}
break;
}
case "finishSecret":
await FinishAsync(e, versions, ct);
break;
default:
throw new ArgumentException($"Unknown step {e.Step}");
}
}
// Step 1: create the new value as AWSPENDING (idempotent: do nothing if it already exists).
private async Task CreateAsync(RotationEvent e, CancellationToken ct)
{
try
{
await secrets.GetSecretValueAsync(new GetSecretValueRequest
{
SecretId = e.SecretId, VersionId = e.ClientRequestToken, VersionStage = Pending,
}, ct);
return; // a retry of this step: the pending value is already there
}
catch (ResourceNotFoundException) { }
var current = await ReadAsync(e.SecretId, Current, ct);
var random = await secrets.GetRandomPasswordAsync(new GetRandomPasswordRequest
{
PasswordLength = 48, ExcludePunctuation = true,
}, ct);
var next = current with { KeyId = NextKeyId(current.KeyId), ApiKey = random.RandomPassword };
await secrets.PutSecretValueAsync(new PutSecretValueRequest
{
SecretId = e.SecretId,
ClientRequestToken = e.ClientRequestToken,
SecretString = JsonSerializer.Serialize(next, Json),
VersionStages = [Pending],
}, ct);
}
// Step 4: move AWSCURRENT to the new version (the old one automatically becomes AWSPREVIOUS),
// then revoke the old key at the partner.
private async Task FinishAsync(RotationEvent e, Dictionary<string, List<string>> versions, CancellationToken ct)
{
var currentVersionId = versions.First(v => v.Value.Contains(Current)).Key;
var old = await ReadAsync(e.SecretId, Current, ct);
await secrets.UpdateSecretVersionStageAsync(new UpdateSecretVersionStageRequest
{
SecretId = e.SecretId,
VersionStage = Current,
MoveToVersionId = e.ClientRequestToken,
RemoveFromVersionId = currentVersionId,
}, ct);
// Clients cache for up to their TTL. Revoking immediately would break them; a real design
// revokes in a later scheduled job after max(cache TTL) has passed. Shown here for completeness.
await partner.RevokeKeyAsync(old.KeyId, ct);
}
private Task<Credential> PendingAsync(RotationEvent e, CancellationToken ct) =>
ReadAsync(e.SecretId, Pending, ct, e.ClientRequestToken);
private async Task<Credential> ReadAsync(string secretId, string stage, CancellationToken ct, string? versionId = null)
{
var value = await secrets.GetSecretValueAsync(new GetSecretValueRequest
{
SecretId = secretId, VersionStage = stage, VersionId = versionId,
}, ct);
return JsonSerializer.Deserialize<Credential>(value.SecretString, Json)
?? throw new InvalidOperationException("Secret is not valid JSON.");
}
private static string NextKeyId(string keyId) =>
int.TryParse(keyId.AsSpan(2), out var n) ? $"k-{n + 1:000}" : "k-001";
public sealed record Credential(string KeyId, string ApiKey);
private sealed class NotConfiguredPartnerAdmin : IPartnerKeyAdmin
{
public Task RegisterKeyAsync(string keyId, string apiKey, CancellationToken ct) => throw new NotSupportedException("Wire the partner client here.");
public Task<bool> TestKeyAsync(string keyId, string apiKey, CancellationToken ct) => throw new NotSupportedException();
public Task RevokeKeyAsync(string keyId, CancellationToken ct) => throw new NotSupportedException();
}
}The rotation function's execution role needs secretsmanager:DescribeSecret, GetSecretValue, PutSecretValue and UpdateSecretVersionStage on this secret, secretsmanager:GetRandomPassword (resource *), kms:Decrypt and kms:GenerateDataKey on the key if it is customer managed, and network access to the partner API and to Secrets Manager.
Code explained line by line
Reading
PartnerApiOptions.SecretId— configuration holds the secret's name, not its value. Names are not sensitive.PartnerApiCredential.ToString()override — if someone logs the object, the key is masked. Defence in depth against accidental logging.new SecretsManagerCache(client, config)withCacheItemTTL— the official cache fromAWSSDK.SecretsManager.Caching. One instance per process (singleton), so the whole app shares it.GetSecretString(secretId, ct)— returns the cachedAWSCURRENTstring, calling Secrets Manager only when the TTL has expired. This cuts both cost (USD 0.05 per 10,000 calls) and latency.RefreshNowAsync— forces an immediate refresh of one secret. Used only after an authentication failure, never per request.PartnerApiKeyHandler— anHttpClientDelegatingHandler, the same pattern as an auth handler in Azure SDK clients. It applies the key and, on 401, refreshes once and retries once.CanRetry— only safe, body-less methods are retried automatically. A payment POST must never be blindly replayed; it needs anIdempotency-Keyheader so the partner can de-duplicate./partner/key-idendpoint — returns the key ID to prove the wiring works without exposing the key. Remove it in production.
Rotation
DescribeSecretAsync+VersionIdsToStages— the guard rails every rotation function needs: rotation enabled, the token is a known version, and it isAWSPENDING(or alreadyAWSCURRENT, in which case a retry is a no-op).createSecret— idempotent: if the pending version already exists (Secrets Manager retries steps), do nothing. OtherwiseGetRandomPassword+PutSecretValuewithVersionStages = ["AWSPENDING"]and the sameClientRequestToken.setSecret— makes the new credential valid at the downstream system (the partner, or the database'sALTER USER).testSecret— proves the new credential works before clients are switched.finishSecret—UpdateSecretVersionStagemovesAWSCURRENTto the new version; Secrets Manager automatically labels the old oneAWSPREVIOUS.- Revoking the old key — only after every client's cache has expired. The comment explains why immediate revocation is dangerous; in production, a scheduled job revokes
AWSPREVIOUSafter the maximum cache TTL. RemainingTime - 2 s— a timeout budget so the function fails cleanly instead of being killed mid-step.
Rotation strategies for databases
- Single user: one database user whose password changes. Simple, but there is a short window where connections opened with the old password fail until clients refresh. Fine with refresh-on-failure logic and connection retry.
- Alternating users: two users (
app_a,app_b); rotation updates the inactive one, then switchesAWSCURRENT. Clients holding the old credentials keep working until the next rotation. Requires a superuser secret for the rotation function. Preferred for high-availability systems.
Security considerations
⚠ Learning shortcut — not for production
The lab encrypts the secret with the AWS managed key aws/secretsmanager, uses an account-scoped resource policy, and the rotation function revokes the old key immediately. In production: a customer managed KMS key whose key policy limits kms:Decrypt to named roles via kms:ViaService = secretsmanager.<region>.amazonaws.com; a resource policy that also requires aws:PrincipalOrgID and, for workloads in VPCs, access through a Secrets Manager VPC endpoint (aws:SourceVpce); a delayed revocation of AWSPREVIOUS; and alarms on rotation failures (EventBridge events from Secrets Manager).
- Least privilege: one secret ARN per statement (with the
-??????suffix wildcard),GetSecretValueonly. Neversecretsmanager:*on*for an application. - Separate duties: humans should not routinely read production secrets. Grant break-glass access through a separate role with MFA and alerting on every
GetSecretValueby a human principal. - CloudTrail:
GetSecretValueis a logged management event. Alert on reads by unexpected principals or from unexpected networks. - No secrets in Terraform state, CDK context, pipeline logs or container images. Generate them in AWS (CDK
GenerateSecretString,get-random-password) so no human sees them. - SCP guard rails: deny
secretsmanager:DeleteSecretwithForceDeleteWithoutRecoveryand denysecretsmanager:PutResourcePolicywithoutBlockPublicPolicyacross the organisation. - Memory hygiene: .NET strings are immutable and cannot be reliably wiped. Keep secrets in as few places as possible, do not copy them into exceptions, logs or metrics, and avoid long-lived static copies outside the cache.
- Cross-Region DR: replicate critical secrets to the recovery Region so failover does not depend on the primary Region being reachable.
Banking use case
Harbourline's Payments Service connects to the card processor with a key pair and to PostgreSQL with a database user. The database secret is the RDS-managed master secret for admin tasks plus an application user rotated with the alternating users strategy every 30 days. The card-processor key rotates every 90 days through a custom C# rotation function very like the one above, which calls the processor's key-management API; the old key is revoked 24 hours later by a scheduled job. Every application reads secrets at runtime through SecretsManagerCache with a 15-minute TTL and refresh-on-401.
The security team gets three dashboards from CloudTrail and EventBridge: rotation success/failure per secret, GetSecretValue by human principals (should be zero outside incidents), and secrets with rotation disabled. The quarterly access review uses IAM Access Analyzer to list every principal that can read each secret. Compliance reviewers still own the decision on whether this meets the bank's key-management policy; the platform provides the evidence.
Common mistakes
- Calling
GetSecretValueon every request → cost, latency, throttling → cache with a TTL. - Caching forever (reading once at start-up into a static) → rotation breaks the app until restart → TTL plus refresh-on-auth-failure.
- Injecting secrets as environment variables and expecting rotation to work → tasks keep the old value → runtime fetch, or force a new deployment after rotation.
- IAM resource ARN without the random suffix →
AccessDenied→ usename-??????or the full ARN. - Customer managed key without granting the app
kms:Decrypt→AccessDeniedExceptionmentioning KMS → add it in the key policy or IAM policy. - Non-idempotent rotation steps → duplicate pending versions or a half-rotated secret after a retry → check for existing
AWSPENDINGfirst. - Revoking the old credential in
finishSecret→ clients with cached values fail for up to one TTL → revoke later. - Logging the secret in an exception message → secret in CloudWatch Logs → mask in
ToString(), never include values in messages.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
AccessDeniedException ... secretsmanager:GetSecretValue | Identity policy ARN mismatch or resource policy deny | Check the -?????? suffix; run validate-resource-policy; check SCPs |
AccessDeniedException ... kms:Decrypt | CMK key policy or IAM lacks Decrypt for the role | Add Decrypt with kms:ViaService condition |
ResourceNotFoundException | Wrong Region or name, or secret scheduled for deletion | Check AWS_REGION; restore-secret if within the recovery window |
| App works, then fails after rotation until restart | Value cached without TTL or never refreshed | Use the cache TTL and refresh-on-401 pattern |
Rotation stuck: AWSPENDING exists on an old version | A previous rotation failed midway | Fix the function; remove the stale AWSPENDING label with update-secret-version-stage; rotate again |
| Rotation Lambda times out | No network path to Secrets Manager or the target (VPC without endpoint/NAT) | Add a Secrets Manager VPC endpoint and a route to the target |
ECS task: ResourceInitializationError: unable to retrieve secret | Execution role lacks GetSecretValue / no network path | Fix execution role (not task role); add endpoints |
ThrottlingException on GetSecretValue | Uncached reads at high request rates | Cache; increase TTL |
Cost and cleanup
What this lesson costs: one secret for a day costs about USD 0.013 (USD 0.40 per month, prorated); a few hundred API calls cost well under one cent. A rotation Lambda runs within the free tier. A customer managed KMS key costs about USD 1 per month prorated ⚠ VERIFY, plus requests. Accounts created after 2025-07-15 have Free Tier credits that cover this. Label: credits (cents).
Free alternative: use a Parameter Store SecureString for the reading part (free) and do the rotation part as a design exercise with the C# function and unit tests only.
Teardown:
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws secretsmanager cancel-rotate-secret --secret-id hl-m03/partner-api 2>/dev/null || true
# Lab only: skip the recovery window so the name can be reused and billing stops now.
aws secretsmanager delete-secret --secret-id hl-m03/partner-api --force-delete-without-recovery
aws lambda delete-function --function-name hl-m03-rotate-partner-key 2>/dev/null || true
# If you created a CMK: schedule deletion (minimum 7 days); it stops being usable immediately when disabled.
# aws kms schedule-key-deletion --key-id alias/hl-m03-secrets --pending-window-in-days 7
aws secretsmanager list-secrets --filters Key=tag-key,Values=Project --query 'SecretList[].Name'Exercises
- 🟢 Guided: Create the secret with the CLI, grant your local SSO role read access, run the demo app, and call
/partner/key-id. Then update the secret value withput-secret-valueand confirm the app shows the new key ID after the cache TTL (set the TTL to 30 seconds for the test). - 🟡 Challenge: Write unit tests for
PartnerApiKeyHandlerwith a fake inner handler that returns 401 for the old key and 200 for the new one, and a fake secret source. Assert exactly one refresh and one retry, and no retry for POST. - 🔴 Live problem: Ticket SEC-3307: "Every 30 days, at 03:00, the Payments API logs about 2 minutes of
28P01 password authentication failederrors and the on-call engineer is paged." The secret uses the single-user rotation strategy and the app reads it at start-up into a static field. Propose a fix and a rollout plan.
Assignment
Write a short secrets standard for your portfolio: classification of secret types, where each may live, delivery pattern per host, rotation period per type, required IAM/KMS/resource-policy controls and monitoring. Include the PartnerApiSecret and PartnerApiKeyHandler classes with tests as the reference implementation.
Solutions
put-secret-value creates a new version labelled AWSCURRENT (the previous becomes AWSPREVIOUS). The app keeps returning the old key ID until the cache item's TTL expires, then the next call fetches the new value. Why: SecretsManagerCache refreshes lazily on access after the TTL; there is no push notification. This is why rotation must keep the old credential valid for at least one TTL.
Create a fake HttpMessageHandler that inspects the X-Api-Key header and returns 401 for old and 200 for new. Use an IPartnerApiSecret-style abstraction (extract an interface from PartnerApiSecret) whose GetAsync returns old until RefreshAsync is called, then new. Wire PartnerApiKeyHandler with InnerHandler = fake and an HttpClient around it. Assertions: for GET, status 200, refresh count 1, inner calls 2; for POST with a body, status 401, refresh count 0, inner calls 1. Why the POST rule: retrying a money movement without an idempotency key can duplicate it.
Root cause: single-user rotation changes the database password; existing pooled connections stay open, but every new connection uses the stale password from the static field until the app restarts or the pool happens to recycle. Fix: (1) read credentials through SecretsManagerCache with a TTL; (2) on authentication failure (28P01), refresh the secret, clear the connection pool (NpgsqlConnection.ClearPool), and retry once; (3) switch to the alternating users strategy so the previous credentials remain valid until the next rotation; (4) build the connection string per connection from the cached secret (Npgsql data sources support a password provider callback ⚠ VERIFY the API in your Npgsql version). Rollout: deploy the code change first (works with both strategies), verify with a manual rotate-secret in staging during business hours, then switch the rotation function to alternating users, and finally adjust paging so a single brief auth error does not page.
Knowledge check
Answer: AWSCURRENT. You can ask for AWSPREVIOUS or AWSPENDING (or a version ID) explicitly.
Answer: createSecret (new value as AWSPENDING), setSecret (apply it to the target system), testSecret (prove it works), finishSecret (move AWSCURRENT to the new version).
Answer: The running task keeps the old value; only new tasks get the new one. Force a new deployment or fetch at runtime.
Answer: For low-risk sensitive configuration that does not need rotation, cross-account resource policies or replication, especially when cost matters. Anything that must rotate or guards customer data belongs in Secrets Manager.
Answer: Secrets Manager appends a hyphen and six random characters to every secret's ARN. A policy using only the name without the suffix (or the full ARN) does not match.
Interview questions
- "How do you manage secrets for a .NET service on AWS?" — Secrets Manager with a CMK and resource policy; the service role can read only its secrets; runtime fetch through
SecretsManagerCachewith a TTL and refresh-on-auth-failure; automatic rotation (managed for RDS, Lambda otherwise); CloudTrail monitoring; no secrets in code, images, env vars or IaC state. - "Explain how rotation works without downtime." — Four steps with staging labels; the new credential is valid before clients switch; clients refresh within a TTL; the old one stays valid (alternating users, delayed revocation) until all caches have expired.
- "Why not environment variables?" — Visible to anything in the process and to describe/debug tooling, captured in dumps and accidental logs, and frozen at start so rotation needs a redeploy.
- "How would you detect secret misuse?" — CloudTrail
GetSecretValueevents by unexpected principals, source IPs or VPC endpoints; Access Analyzer for external access; EventBridge alerts on rotation failures and policy changes.
Certification practice
A .NET application on Lambda connects to an Amazon RDS for PostgreSQL database. Credentials must rotate automatically every 30 days, and the application must not be redeployed for rotations. Which solution meets these requirements with the least operational effort?
A. Store the password in a Lambda environment variable and update it every 30 days with a pipeline.
B. Store the credentials in Secrets Manager with rotation enabled, and retrieve them at runtime with a client-side cache.
C. Store the password in Parameter Store SecureString and create an EventBridge rule to change it.
D. Hard-code the password and rotate the database user manually.
Answer: B. Secrets Manager rotates on a schedule (managed or Lambda rotation) and runtime retrieval with caching picks up new values without redeployment. A needs a redeploy and exposes the value. C has no built-in rotation. D is insecure and manual.
An API running on 30 Fargate tasks calls GetSecretValue on every request, about 2,000 requests per second. The team sees ThrottlingException errors and a high Secrets Manager bill. What is the best fix?
A. Request a quota increase for GetSecretValue.
B. Move the secret into an environment variable.
C. Use the Secrets Manager client-side caching library with a TTL in each task.
D. Create a secret per task.
Answer: C. Caching reduces calls to roughly one per TTL per task, removing throttling and cost while still picking up rotations. A treats the symptom and keeps the cost. B loses rotation support and exposes the value. D multiplies secrets and cost.
A security engineer must ensure that a Secrets Manager secret encrypted with a customer managed KMS key can be decrypted only when the request comes through Secrets Manager, even if an IAM policy grants broader KMS permissions. What should the engineer add to the KMS key policy?
A. A condition kms:ViaService equal to secretsmanager.us-east-1.amazonaws.com on the kms:Decrypt permission.
B. A condition aws:SecureTransport equal to true.
C. A kms:GrantIsForAWSResource condition on kms:CreateGrant only.
D. Enable automatic key rotation.
Answer: A. kms:ViaService limits key use to requests made by the named service on the principal's behalf. B only requires TLS. C applies to grants, not direct decryption. D rotates key material but does not restrict usage.
Completion checklist
- I can explain why environment variables, config files and IaC state are poor places for secrets
- I created a secret with a resource policy and read it through the cache
- I can describe the four rotation steps and the three staging labels
- My client code survives a rotation without a restart
- I force-deleted the lab secret and checked nothing remains
Key takeaways
- Secrets Manager for anything that rotates or guards customer data;
SecureStringonly for low-risk, non-rotating values. - Read secrets at runtime through a cache with a TTL; refresh once on authentication failure.
- Rotation is createSecret → setSecret → testSecret → finishSecret, driven by staging labels; keep the old credential valid for at least one client TTL.
- Three locks: IAM, the secret resource policy and the KMS key policy; watch access with CloudTrail.
- Injected secrets (ECS, Beanstalk) are frozen at start; plan redeployments or prefer runtime fetching.
Further reading
- Rotate AWS Secrets Manager secrets
- Rotation by Lambda function
- Rotation schedules
- Pass Secrets Manager secrets through Amazon ECS environment variables
- Fetching secrets and parameters to Elastic Beanstalk environment variables
- Using Parameter Store parameters in AWS Lambda functions (Parameters and Secrets Lambda Extension)
- AWS Secrets Manager pricing
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Configuration done right
Layer ASP.NET Core configuration from appsettings.json through environment variables to Systems Manager Parameter Store and AWS AppConfig, validate it at start-up, reload it safely, and roll out feature flags gradually with automatic rollback.
Health checks that help instead of hurt
Design liveness and readiness endpoints in ASP.NET Core, wire them correctly to ALB target groups, ECS container health checks and Elastic Beanstalk, tune timings and grace periods, shut down gracefully, and avoid the restart loop that turns a database blip into a full outage.