Secrets in CI/CD: keep them out of pipelines, logs and images
Classify every secret a delivery pipeline touches, remove the ones OIDC makes unnecessary, keep runtime secrets out of the pipeline entirely, and handle the rest with GitHub environment secrets, AWS Secrets Manager, CodeBuild and BuildKit secret mounts so they never appear in logs or image layers.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager 4.1.1.2) · Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3 · GitHub Actions (aws-actions/aws-secretsmanager-get-secrets v4.0.0, aws-actions/configure-aws-credentials v6.3.0) · Docker BuildKit · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Sort the secrets a pipeline touches into four kinds — cloud credentials, build-time secrets, deploy-time references and runtime secrets — and choose the right home for each.
- Explain what GitHub repository, organisation and environment secrets protect, and where their masking stops working.
- Fetch a secret from AWS Secrets Manager inside a GitHub Actions job through an OIDC role, and inside AWS CodeBuild through the buildspec.
- Pass a private NuGet feed token to
docker buildwith a BuildKit secret mount so it never lands in an image layer. - Deliver database credentials to an ASP.NET Core service on ECS without the pipeline ever seeing them, and survive a rotation with a small C# credential provider.
- Respond to a leaked secret in the right order: revoke, rotate, investigate, then clean up history.
Prerequisites
- GitHub Actions with OIDC.
- Environment separation — GitHub environments and per-environment roles.
- Module 3: Configuration done right — configuration providers and secrets in .NET.
- Pipeline security scanning — secrets scanning with Gitleaks.
- Tools: AWS CLI v2 with the
learn-sandboxprofile, Docker with BuildKit (default in current Docker Desktop), .NET 10 SDK (or Docker).
Why this matters in banking
Three real patterns, each of which has caused public incidents across the industry, re-told with Harbourline names:
- The key in the pipeline. A team stored an IAM user's access key as a repository secret so the pipeline could deploy. A compromised third-party action in that workflow printed environment variables into the build log of a public repository. Within minutes, automated scanners had the key; within an hour, crypto-mining instances were running in the account. With OIDC there would have been no key to steal.
- The token in the image. A Dockerfile used
ARG NUGET_TOKENto restore packages from the bank's private feed. Build arguments are recorded in the image history. Anyone who could pull the image — including a vendor with read access to the registry — could read the token withdocker history. - The password in the log. A deployment script ran
set -xfor debugging and echoed a connection string built from a database password. GitHub masked the password, but the script had URL-encoded it first, and the encoded form was not masked.
For a bank these are not just hygiene issues. PCI DSS requires strong protection and restricted access for credentials to systems in scope; GDPR / UK GDPR breaches often start with a leaked credential; and auditors will ask how secrets are stored, who can read them, and how quickly they can be rotated. A good answer is short: "the pipeline holds no long-term credentials; runtime secrets never pass through it; the few build-time secrets are scoped to an environment, short-lived where possible, and kept out of logs and images by design."
The idea in plain language
A secret is any value that grants access: a password, an API token, a private key, a signing key, a connection string with a password in it. A delivery pipeline is dangerous for secrets because it runs lots of other people's code (actions, NuGet packages, build tools) and writes long, searchable logs.
The safest secret is one the pipeline never has. So the approach is to eliminate, then isolate, then protect:
| Kind | Example | Where it should live | Does the pipeline see it? |
|---|---|---|---|
| Cloud credentials | AWS access keys for deploying | Nowhere — replaced by OIDC federation and short-lived role sessions | Only a short-lived session, never a stored key |
| Build-time secrets | Private NuGet feed token, SAST tool API token | GitHub environment secret or Secrets Manager, fetched for one step | Yes, briefly — so keep it scoped and out of logs and images |
| Deploy-time references | "The DB password is in secret hl/accounts/db" | In the IaC or task definition as an ARN, not a value | Only the reference |
| Runtime secrets | Database password, partner API key, card tokenisation key | Secrets Manager (or KMS for keys), read by the running service's role | No |
| Signing keys | Image or artefact signing key | AWS KMS / AWS Signer, key never exported | No — the pipeline asks the service to sign |
Analogy: a cash-in-transit courier (the pipeline) carries sealed bags between branches. The courier never knows the vault combination (runtime secret): each branch's vault opens for that branch's staff only. The courier does carry a delivery pass (build-time secret), but it is valid for one route, one day, and it is never photocopied into the delivery log.
What problem the AWS services and tools solve
| Tool | What it gives you | Limits and sharp edges |
|---|---|---|
| GitHub Actions secrets (repository, organisation, environment) | Encrypted values injected into jobs; automatically masked in logs | 48 KB per secret; up to 100 repository and 100 environment secrets. Masking matches exact strings only — transformed values (base64, URL-encoded, split) are not masked. Avoid structured (JSON/YAML) secret values |
| Environment secrets | Available only to jobs that reference the environment, and only after its protection rules (e.g. required reviewers) pass | Administrators can bypass protection unless that is disabled for the environment |
| Fork protection | Secrets (other than GITHUB_TOKEN) are not passed to workflows triggered by pull requests from forks | pull_request_target and workflow_run run with secrets in the base repository's context — dangerous with untrusted code |
| AWS Secrets Manager | Encrypted storage with KMS, fine-grained IAM, resource policies, versioning, automatic rotation, CloudTrail audit of every read | $0.40 per secret per month plus API calls ⚠ VERIFY current pricing. Callers must cache to avoid cost and throttling |
SSM Parameter Store SecureString | Cheaper storage for configuration with some secret values | No built-in rotation; standard parameters are free, advanced ones are not |
aws-actions/aws-secretsmanager-get-secrets | Fetches secrets into masked environment variables in a GitHub job | Values are only as safe as the steps that run after it |
CodeBuild env.secrets-manager | Injects secrets into the build container from Secrets Manager | The CodeBuild service role needs secretsmanager:GetSecretValue on exactly those secrets |
ECS task definition secrets | The ECS agent fetches the secret at task start and sets it as an environment variable in the container | Retrieved at start only; a rotated value is picked up when tasks are replaced |
BuildKit --secret | Mounts a secret as a file (or env) for one RUN instruction only; nothing in layers or history | Requires BuildKit (default in current Docker) |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS / GitHub equivalent | What's different |
|---|---|---|
| Azure DevOps secret variables | GitHub Actions secrets | Similar masking behaviour and similar blind spots for transformed values |
| Variable groups linked to Key Vault | aws-secretsmanager-get-secrets in a job, or CodeBuild env.secrets-manager | Access is through the job's OIDC role; scope per GitHub environment |
| Service connection with a client secret | OIDC federation to an IAM role | No secret to store or rotate |
| Key Vault references in App Service settings | ECS task definition secrets with valueFrom an ARN | The platform resolves the secret at start; the deployment carries only the reference |
| Managed identity reading Key Vault at runtime | ECS task role / Lambda execution role reading Secrets Manager | Same pattern; cache in process |
| Key Vault secret rotation with Event Grid | Secrets Manager rotation (Lambda-based, or managed for RDS master passwords) | RDS can manage the master user password in Secrets Manager for you |
Docker --build-arg secrets in Azure Pipelines | BuildKit --secret | Build args persist in image metadata; secret mounts do not |
Architecture
The important thing is what is missing: there is no arrow from Secrets Manager to the GitHub job. The pipeline deploys a task definition that names the secret; ECS fetches the value at runtime with the task execution role.
Security-boundary view
Step-by-step implementation
Inventory every secret the pipeline touches
Search the workflows for secrets. and for anything that looks like a credential in variables. For each, record: what it grants, who owns it, how it is rotated, and which of the four kinds it is. Most teams find at least one stored AWS key and one runtime secret that does not need to be there.
Eliminate cloud credentials
Replace every stored AWS access key with an OIDC role per environment. Delete the IAM user and its keys afterwards — leaving them "just in case" defeats the purpose.
Move runtime secrets out of the pipeline
Create the secret in Secrets Manager (or let RDS manage the master password). Reference its ARN in the ECS task definition secrets block or read it at runtime from Lambda. Remove the value from GitHub. Grant secretsmanager:GetSecretValue on that ARN to the task execution role (for injection) or task role (for runtime reads) — not to the deploy role.
Scope the build-time secrets that remain
Store them as environment secrets, not repository secrets, so they reach only the jobs that declare that environment. Prefer short-lived tokens: for example, AWS CodeArtifact authorisation tokens obtained with the OIDC role (valid for up to 12 hours) instead of a long-lived feed password.
Keep secrets out of images
Never use ARG or ENV for secrets in a Dockerfile. Use RUN --mount=type=secret so the secret exists only for that step. Use a multi-stage build so that tools and config files used during restore are not copied into the runtime image.
Keep secrets out of logs
Pass secrets to commands through environment variables, not command-line arguments. Never set -x in a step that handles secrets. If you derive a new value from a secret (an encoded form, a generated token), register it with ::add-mask:: before anything prints it. Don't upload workspace folders that may contain credentials files as artefacts.
Detect and respond
Turn on GitHub secret scanning and push protection where your plan allows it, and run Gitleaks in CI (see Pipeline security scanning). Write the leak runbook: revoke first, rotate, check CloudTrail for use of the leaked credential, then clean history.
AWS Console instructions
Create a secret:
- Console → Secrets Manager → Store a new secret.
- Other type of secret; add key/value pairs
usernameandpassword(or let RDS manage the master password — see below). - Encryption key: choose a customer managed KMS key (for example
alias/hl-accounts-secrets) so you control the key policy. - Name:
hl/accounts/db. Add tagsProject=cea-lab,Lab=m09-l11,Environment=learn. - On the rotation page, enable rotation if you have a rotation function; otherwise leave it for now.
Let RDS manage the database master password:
- Console → RDS → Databases → Modify (or Create database).
- Under Credentials management, choose Managed in AWS Secrets Manager. RDS creates and rotates the secret; your application reads it like any other.
GitHub environment secret:
- GitHub → repository → Settings → Environments →
prod→ Environment secrets → Add secret. - Name
NUGET_FEED_TOKEN; paste the token. It is now available only to jobs withenvironment: prod, and only after approval.
Check who read a secret:
- Console → CloudTrail → Event history, filter Event name = GetSecretValue, and look at the
userIdentityandrequestParameters.secretIdof each event.
AWS CLI
P=learn-sandbox
# 1. Create a secret encrypted with a customer managed key, with lab tags.
# The value is read from a file so it never appears in shell history or process lists.
aws secretsmanager create-secret --name hl/accounts/db \
--kms-key-id alias/hl-accounts-secrets \
--secret-string file://db-secret.json \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m09-l11 Key=Environment,Value=learn \
--profile $P
shred -u db-secret.json 2>/dev/null || rm -f db-secret.json
# 2. Restrict the secret with a resource policy: only the task execution role may read it.
aws secretsmanager put-resource-policy --secret-id hl/accounts/db \
--resource-policy file://secret-policy.json --block-public-policy --profile $P
# 3. Validate the resource policy before attaching changes (Access Analyzer-backed check).
aws secretsmanager validate-resource-policy --secret-id hl/accounts/db \
--resource-policy file://secret-policy.json --profile $P
# 4. See metadata only (never the value) — safe to run in pipelines.
aws secretsmanager describe-secret --secret-id hl/accounts/db \
--query '{Name:Name,Rotation:RotationEnabled,LastRotated:LastRotatedDate,Kms:KmsKeyId}' --profile $P
# 5. Rotate now (requires a rotation function or an RDS-managed secret).
aws secretsmanager rotate-secret --secret-id hl/accounts/db --profile $P
# 6. Who has read this secret in the last day? (CloudTrail management events)
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
--start-time "$(date -u -d '-1 day' +%Y-%m-%dT%H:%M:%SZ)" \
--query 'Events[].[EventTime,Username]' --output table --profile $P
# 7. A short-lived CodeArtifact token instead of a stored feed password
# (run inside the job after configure-aws-credentials; valid up to 12 hours).
aws codeartifact get-authorization-token --domain harbourline --domain-owner 111122223333 \
--duration-seconds 3600 --query authorizationToken --output text --profile $PThe resource policy (file secret-policy.json) denies everyone except the task execution role, even principals in the same account with broad IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OnlyAccountsApiExecutionRole",
"Effect": "Deny",
"Principal": "*",
"Action": "secretsmanager:GetSecretValue",
"Resource": "*",
"Condition": {
"ArnNotEquals": {
"aws:PrincipalArn": [
"arn:aws:iam::111122223333:role/hl-accounts-api-exec",
"arn:aws:iam::111122223333:role/hl-break-glass"
]
}
}
}
]
}Infrastructure as Code
{
"family": "hl-accounts-api",
"executionRoleArn": "arn:aws:iam::111122223333:role/hl-accounts-api-exec",
"taskRoleArn": "arn:aws:iam::111122223333:role/hl-accounts-api-task",
"containerDefinitions": [
{
"name": "api",
"image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/hl-accounts-api@sha256:REPLACED_BY_PIPELINE",
"environment": [
{ "name": "Database__Host", "value": "hl-accounts.cluster-abc.us-east-1.rds.amazonaws.com" },
{ "name": "Database__Name", "value": "accounts" }
],
"secrets": [
{ "name": "Database__Username",
"valueFrom": "arn:aws:secretsmanager:us-east-1:111122223333:secret:hl/accounts/db-AbCdEf:username::" },
{ "name": "Database__Password",
"valueFrom": "arn:aws:secretsmanager:us-east-1:111122223333:secret:hl/accounts/db-AbCdEf:password::" }
]
}
]
}The valueFrom format is secret-arn:json-key:version-stage:version-id; the two trailing colons mean "current version". The execution role needs secretsmanager:GetSecretValue on the secret and kms:Decrypt on its key.
Complete code
1. The GitHub Actions job: environment secret, Secrets Manager, masking
name: build-image
on:
push:
branches: [main]
permissions:
contents: read
id-token: write
jobs:
build:
runs-on: ubuntu-24.04
environment: build # environment secrets below exist only for this environment
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false # do not leave GITHUB_TOKEN in .git/config for later steps
- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
with:
role-to-assume: ${{ vars.BUILD_ROLE_ARN }}
aws-region: us-east-1
# Fetch a third-party scanner token from Secrets Manager into a masked env var.
- uses: aws-actions/aws-secretsmanager-get-secrets@e952bd7125c27e5be0060e50955b90a7e5addd99 # v4.0.0
with:
secret-ids: |
SCANNER_TOKEN, hl/ci/scanner-token
# A derived value must be masked explicitly BEFORE anything can print it.
- name: Get a short-lived feed token
run: |
TOKEN=$(aws codeartifact get-authorization-token --domain harbourline \
--domain-owner "${{ vars.ARTIFACT_ACCOUNT_ID }}" --duration-seconds 1800 \
--query authorizationToken --output text)
echo "::add-mask::$TOKEN"
echo "FEED_TOKEN=$TOKEN" >> "$GITHUB_ENV"
# The token reaches docker build as a BuildKit secret, not a build argument.
- name: Build image
run: |
docker build \
--secret id=nuget_token,env=FEED_TOKEN \
-t hl-accounts-api:${GITHUB_SHA} .
# Prove the token is not in the image history (fails the job if it is).
- name: Check image history for the token
run: |
if docker history --no-trunc hl-accounts-api:${GITHUB_SHA} | grep -F -q "$FEED_TOKEN"; then
echo "::error::Feed token found in image history"; exit 1
fi2. Dockerfile with a secret mount and a clean runtime stage
# syntax=docker/dockerfile:1
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY nuget.config ./
COPY src/Harbourline.Accounts.Api/*.csproj src/Harbourline.Accounts.Api/
# The secret is mounted at /run/secrets/nuget_token for THIS step only.
# NuGet expands %NUGET_TOKEN% from nuget.config at restore time.
RUN --mount=type=secret,id=nuget_token \
NUGET_TOKEN="$(cat /run/secrets/nuget_token)" \
dotnet restore src/Harbourline.Accounts.Api/Harbourline.Accounts.Api.csproj
COPY . .
RUN dotnet publish src/Harbourline.Accounts.Api -c Release -o /app --no-restore
# The runtime stage copies only the published output: no nuget.config, no SDK, no token.
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
WORKDIR /app
COPY --from=build /app .
USER $APP_UID
ENTRYPOINT ["dotnet", "Harbourline.Accounts.Api.dll"]<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
<add key="harbourline" value="https://harbourline-111122223333.d.codeartifact.us-east-1.amazonaws.com/nuget/internal/v3/index.json" />
</packageSources>
<packageSourceCredentials>
<harbourline>
<add key="Username" value="aws" />
<!-- Expanded from the environment at restore time; the file holds no secret. -->
<add key="ClearTextPassword" value="%NUGET_TOKEN%" />
</harbourline>
</packageSourceCredentials>
</configuration>3. A rotation-aware database credential provider in C#
When the secret is injected by ECS, a rotation is only picked up when tasks restart. For services that must survive rotation without a restart, read the secret at runtime, cache it, and refresh once when the database rejects the password.
using System.Text.Json;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;
namespace Harbourline.Accounts.Api.Data;
public sealed record DbCredentials(string Username, string Password);
public interface IDbCredentialProvider
{
Task<DbCredentials> GetAsync(CancellationToken ct);
/// <summary>Forget the cached value, e.g. after the database rejected the password.</summary>
void Invalidate();
}
public sealed class SecretsManagerDbCredentialProvider(
IAmazonSecretsManager secrets, string secretId, TimeProvider clock, TimeSpan maxAge) : IDbCredentialProvider
{
private readonly SemaphoreSlim _gate = new(1, 1);
private DbCredentials? _cached;
private DateTimeOffset _fetchedAt;
public async Task<DbCredentials> GetAsync(CancellationToken ct)
{
if (_cached is { } fresh && clock.GetUtcNow() - _fetchedAt < maxAge) return fresh;
await _gate.WaitAsync(ct); // one refresh at a time, even under load
try
{
if (_cached is { } c && clock.GetUtcNow() - _fetchedAt < maxAge) return c;
var response = await secrets.GetSecretValueAsync(
new GetSecretValueRequest { SecretId = secretId }, ct);
using var doc = JsonDocument.Parse(response.SecretString);
_cached = new DbCredentials(
doc.RootElement.GetProperty("username").GetString()!,
doc.RootElement.GetProperty("password").GetString()!);
_fetchedAt = clock.GetUtcNow();
return _cached;
}
finally
{
_gate.Release();
}
}
public void Invalidate() => _cached = null;
// Never let the password reach logs through ToString() or a debugger display.
public override string ToString() => $"SecretsManagerDbCredentialProvider({secretId})";
}using Npgsql;
namespace Harbourline.Accounts.Api.Data;
public sealed class ResilientConnectionFactory(IDbCredentialProvider credentials, string host, string database)
{
// PostgreSQL SQLSTATE 28P01 = invalid_password.
private const string InvalidPassword = "28P01";
public async Task<NpgsqlConnection> OpenAsync(CancellationToken ct)
{
try
{
return await OpenOnceAsync(ct);
}
catch (PostgresException ex) when (ex.SqlState == InvalidPassword)
{
// The secret was probably rotated since we cached it: refresh once and retry.
credentials.Invalidate();
return await OpenOnceAsync(ct);
}
}
private async Task<NpgsqlConnection> OpenOnceAsync(CancellationToken ct)
{
var c = await credentials.GetAsync(ct);
var csb = new NpgsqlConnectionStringBuilder
{
Host = host,
Database = database,
Username = c.Username,
Password = c.Password,
SslMode = SslMode.VerifyFull // TLS with certificate and host-name verification
};
var conn = new NpgsqlConnection(csb.ConnectionString);
await conn.OpenAsync(ct);
return conn;
}
}using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;
using Harbourline.Accounts.Api.Data;
using Microsoft.Extensions.Time.Testing;
using NSubstitute;
using Xunit;
public class DbCredentialProviderTests
{
private static IAmazonSecretsManager SecretsReturning(params string[] passwords)
{
var sm = Substitute.For<IAmazonSecretsManager>();
var responses = passwords
.Select(p => new GetSecretValueResponse { SecretString = $$"""{"username":"accounts_app","password":"{{p}}"}""" })
.ToArray();
sm.GetSecretValueAsync(Arg.Any<GetSecretValueRequest>(), Arg.Any<CancellationToken>())
.Returns(responses[0], responses.Skip(1).ToArray());
return sm;
}
[Fact]
public async Task Caches_within_max_age()
{
var sm = SecretsReturning("first", "second");
var clock = new FakeTimeProvider();
var provider = new SecretsManagerDbCredentialProvider(sm, "hl/accounts/db", clock, TimeSpan.FromMinutes(15));
await provider.GetAsync(CancellationToken.None);
clock.Advance(TimeSpan.FromMinutes(5));
var second = await provider.GetAsync(CancellationToken.None);
Assert.Equal("first", second.Password);
await sm.Received(1).GetSecretValueAsync(Arg.Any<GetSecretValueRequest>(), Arg.Any<CancellationToken>());
}
[Fact]
public async Task Invalidate_forces_a_fresh_read_after_rotation()
{
var sm = SecretsReturning("before-rotation", "after-rotation");
var provider = new SecretsManagerDbCredentialProvider(sm, "hl/accounts/db", new FakeTimeProvider(), TimeSpan.FromHours(1));
await provider.GetAsync(CancellationToken.None);
provider.Invalidate();
var refreshed = await provider.GetAsync(CancellationToken.None);
Assert.Equal("after-rotation", refreshed.Password);
}
[Fact]
public void ToString_never_contains_the_password()
{
var provider = new SecretsManagerDbCredentialProvider(
SecretsReturning("s3cr3t"), "hl/accounts/db", new FakeTimeProvider(), TimeSpan.FromMinutes(1));
Assert.DoesNotContain("s3cr3t", provider.ToString());
}
}The test project references xunit.v3 4.0.3, NSubstitute and Microsoft.Extensions.TimeProvider.Testing (for FakeTimeProvider). Register the provider in Program.cs with builder.Services.AddAWSService<IAmazonSecretsManager>() and a singleton SecretsManagerDbCredentialProvider, so all requests share one cache.
Code explained line by line
Workflow
environment: build— environment secrets and theBUILD_ROLE_ARNvariable are only available to this job.persist-credentials: false—actions/checkoutotherwise stores theGITHUB_TOKENin the local Git config, where every later step (including third-party actions) can read it.aws-secretsmanager-get-secretswithSCANNER_TOKEN, hl/ci/scanner-token— fetches the secret through the job's OIDC role and exposes it as a masked environment variable namedSCANNER_TOKEN.echo "::add-mask::$TOKEN"— the CodeArtifact token was not a GitHub secret, so GitHub does not know to hide it. Masking must happen before any step could print it.echo "FEED_TOKEN=$TOKEN" >> "$GITHUB_ENV"— passes the value to later steps as an environment variable, not as a step output, which would be visible to other jobs.--secret id=nuget_token,env=FEED_TOKEN— BuildKit reads the value from the environment variable and offers it to the build as a secret mount.- The history check is a cheap regression test: if anyone reintroduces
ARG NUGET_TOKEN, the job fails.
Dockerfile
# syntax=docker/dockerfile:1— selects a modern Dockerfile frontend that supports secret mounts.RUN --mount=type=secret,id=nuget_token— the file/run/secrets/nuget_tokenexists only while thisRUNexecutes; it is not in any layer.NUGET_TOKEN="$(cat ...)" dotnet restore— sets the variable for that one command only.- The runtime stage copies
/apponly;nuget.config, the SDK and any restore caches stay in the discarded build stage. USER $APP_UID— the official .NET images define a non-root user; run as it.
SecretsManagerDbCredentialProvider
- Primary-constructor parameters (
secrets,secretId,clock,maxAge) —TimeProvidermakes time testable. - Fast path returns the cached value without locking when it is fresh enough.
SemaphoreSlim— when the cache expires under load, only one request calls Secrets Manager; the others wait and reuse the result (avoids a burst of API calls and throttling).- Double-check inside the lock — another caller may have refreshed while this one waited.
JsonDocument.Parse(response.SecretString)— RDS-managed and most manual secrets storeusernameandpasswordas JSON keys.Invalidate()— called when the database says the password is wrong.ToString()override — prevents accidental logging of internals.
ResilientConnectionFactory
SqlState == "28P01"— PostgreSQL's code for an invalid password, the signal that rotation happened.- Retry once: a second failure is a real problem and must surface, not loop.
SslMode.VerifyFull— encrypt in transit and verify the server's certificate and host name; credentials should never cross the network in clear.
Security considerations
⚠ Learning shortcut — not for production
In this lesson the secret's resource policy names a hl-break-glass role so you can still read it while experimenting. In production, the break-glass path should require MFA through IAM Identity Center, raise an alert on every use, and be reviewed afterwards — reading a production database password is a significant event.
⚠ Learning shortcut — not for production
The lab secret uses a 15-minute in-process cache and no automatic rotation. In production, enable rotation (or RDS-managed passwords), choose a cache age shorter than the rotation's overlap window, and test a rotation in staging before relying on it.
- Masking is a safety net, not a control. It only matches exact strings. Base64, URL-encoding, JSON-escaping, splitting across lines or printing character by character all bypass it. Design so secrets are never printed in any form.
- Structured secrets mask badly. A JSON secret stored in GitHub may not be redacted correctly; store one value per secret.
pull_request_targetandworkflow_runrun with the base repository's secrets. Never check out and build untrusted pull-request code in those workflows.- Self-hosted runners keep state between jobs unless they are ephemeral. A secret written to disk by one job can be read by the next. Use ephemeral runners (or GitHub-hosted runners) for anything that handles secrets.
- Terraform state and CloudFormation templates can contain secret values. Prefer generated or managed secrets; never put a secret in a CloudFormation parameter without
NoEcho, and rememberNoEchodoes not hide values that you reference elsewhere in outputs or metadata. - Third-party actions see your environment. Every action in a job that has a secret can read it. Pin actions by commit SHA and keep the secret in as few steps (ideally jobs) as possible — see Pipeline security.
Banking use case
Harbourline's secrets inventory for the Payments Service pipeline, before and after this lesson's changes:
| Secret | Before | After |
|---|---|---|
| AWS deploy credentials | IAM user key in repository secret | Removed — OIDC role per environment |
| Database password | Repository secret, passed to aws ecs register-task-definition as an environment value | Secrets Manager (RDS-managed, rotated); task definition holds only the ARN |
| Sanctions-screening vendor API key | Repository secret, read by integration tests and by the app | Two keys: a sandbox key as a test environment secret for tests; the production key in Secrets Manager, read only by the running service |
| Private NuGet feed password | Organisation secret, passed as --build-arg | CodeArtifact token from the OIDC role, BuildKit secret mount |
| SAST tool token | Repository secret | build environment secret |
| Image signing key | Private key file in a repository secret | AWS Signer / ECR managed signing — no key leaves AWS |
The before-state had six long-lived secrets readable by any workflow on any branch. The after-state has two scoped, short-lived or managed secrets in GitHub; nothing in GitHub can read customer data.
Common mistakes
- Passing secrets as command-line arguments → visible in process listings and sometimes in logs → pass by environment variable or file.
ARG/ENVfor secrets in Dockerfiles → stored in image metadata forever → BuildKit secret mounts.- Debugging with
set -x→ every expanded variable is printed, encoded forms are not masked → never in steps that touch secrets. - Uploading the whole workspace as an artefact → credentials files and
.envfiles travel with it → upload explicit paths only. - Deploy role can read runtime secrets → a compromised pipeline can read the database password → only runtime roles get
GetSecretValue. - Rotating without testing the application → outage at the first automatic rotation → rehearse rotation in staging; implement refresh-on-auth-failure.
- Cleaning Git history before revoking a leaked secret → the secret is still valid and still in forks, caches and clones → revoke first, always.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
ECS task stops with ResourceInitializationError: unable to pull secrets | Execution role lacks GetSecretValue or kms:Decrypt, or no network path to Secrets Manager from private subnets | Add permissions; add a VPC endpoint for Secrets Manager (or NAT) |
| Secret appears in the log in an encoded form | Masking only matches exact values | Mask the derived value with ::add-mask:: first; better, stop printing it |
AccessDeniedException reading a secret despite an allow in IAM | Explicit deny in the secret's resource policy, or KMS key policy does not allow the role | Check the resource policy and the key policy |
| Secrets empty in a pull-request workflow | PR is from a fork; secrets are withheld by design | Run secret-dependent jobs only after merge, or with approval for first-time contributors; do not switch to pull_request_target |
App fails after rotation with 28P01 | Cached old password; no refresh path | Add invalidate-and-retry; replace tasks if using ECS-injected secrets |
docker build error secret nuget_token not found | Build run without --secret or BuildKit disabled | Pass --secret id=...; use a current Docker with BuildKit |
| Environment secret not available in a job | Job does not declare environment:, or approval pending | Add environment: <name>; check the run's pending deployments |
Cost and cleanup
Cost: GitHub secrets are free. A Secrets Manager secret costs about $0.40 per month (pro-rated) plus $0.05 per 10,000 API calls ⚠ VERIFY current pricing; new secrets may have a free trial period. A customer managed KMS key costs about $1 per month (pro-rated) plus requests. If you keep a secret and a key for one day and delete them, the cost is a few cents. RDS (Terraform tab) is not needed for this lesson — skip it, or deploy and destroy within an hour. Label: free (within cents).
P=learn-sandbox
# Delete the secret immediately (no recovery window) — lab only.
aws secretsmanager delete-secret --secret-id hl/accounts/db --force-delete-without-recovery --profile $P
aws secretsmanager delete-secret --secret-id hl/ci/scanner-token --force-delete-without-recovery --profile $P
# Schedule the KMS key for deletion (minimum waiting period is 7 days; billing stops when deleted).
KEY=$(aws kms describe-key --key-id alias/hl-accounts-secrets --query KeyMetadata.KeyId --output text --profile $P)
aws kms delete-alias --alias-name alias/hl-accounts-secrets --profile $P
aws kms schedule-key-deletion --key-id "$KEY" --pending-window-in-days 7 --profile $P
# Confirm nothing tagged for this lesson remains (scheduled-for-deletion keys may still show).
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m09-l11 \
--query 'ResourceTagMappingList[].ResourceARN' --profile $PExercises
- 🟢 Guided: Build the Dockerfile above twice: once with
ARG NUGET_TOKENand--build-arg, once with the secret mount. Rundocker history --no-truncon both and find the token in the first. Then create a secret in Secrets Manager and fetch it in a GitHub job withaws-secretsmanager-get-secrets; tryecho "$SCANNER_TOKEN"andecho "$SCANNER_TOKEN" | base64and compare what the log shows. - 🟡 Challenge: Write an ECS task definition and IAM policies for the Accounts API where the deploy role can register the task definition and pass the roles, but cannot read the database secret. Prove it with
aws iam simulate-principal-policyagainst both roles. - 🔴 Live problem: At 14:05 a developer notices that a sanctions-vendor API key appears in a public GitHub Actions log from yesterday. The key is used by the production Payments Service. Write the incident timeline for the next two hours: who you call, what you revoke and in which order, how you keep payments flowing, what you check in logs, and what you change so it cannot recur.
Assignment
Add a SECRETS.md to your portfolio repository with: an inventory table (secret, kind, store, scope, rotation, owner); a diagram showing which roles can read which secrets; the leak runbook; and evidence (screenshots or command output) that your image history contains no secrets and that your deploy role cannot read runtime secrets.
Solutions
With ARG, docker history --no-trunc shows the RUN instruction with NUGET_TOKEN=<value> in it — anyone who can pull the image can read the token. With the secret mount, the history shows only RUN --mount=type=secret,id=nuget_token ... and no value. In the GitHub log, echo "$SCANNER_TOKEN" prints ***, but the base64 version prints the encoded token in full. Why: masking is string matching on the exact value; transformations escape it. The fix is to never print secrets, and to add-mask any derived value that might be printed.
The deploy role gets ecs:RegisterTaskDefinition, ecs:UpdateService on the service, and iam:PassRole on the task and execution roles with iam:PassedToService = ecs-tasks.amazonaws.com. It gets no secretsmanager:*. The execution role gets secretsmanager:GetSecretValue on the one secret ARN and kms:Decrypt on its key. simulate-principal-policy --action-names secretsmanager:GetSecretValue --resource-arns <secret-arn> returns implicitDeny for the deploy role and allowed for the execution role. Add the secret resource policy from this lesson and the deploy role is explicitly denied even if someone later gives it a broad IAM policy. Why: passing a role is not the same as being able to use it; the pipeline can arrange for ECS to fetch the secret without ever holding it.
14:05–14:15: declare a security incident; inform the security operations team and the payments service owner. Revoke first: ask the vendor (or use its console) to issue a new key and revoke the leaked one. To keep payments flowing, put the new key into Secrets Manager before revoking if the vendor allows two active keys; otherwise accept a short screening outage — payments should queue, not bypass screening. 14:15–15:00: update the secret; force new ECS tasks (or rely on the refresh path); confirm screening calls succeed. Check the vendor's access logs for use of the leaked key from unknown IPs since yesterday; check GitHub audit log for who viewed the run. 15:00–16:00: find how it leaked (which step printed it and why masking failed); delete the log; then fix the cause: remove the production key from the pipeline entirely (tests should use a sandbox key), add add-mask where values are derived, and add a Gitleaks rule. Record everything for the post-incident review; whether external notification is required is a decision for security, legal and compliance.
Knowledge check
Answer: It is only available to jobs that declare that environment, and only after the environment's protection rules (such as required reviewers and branch restrictions) pass. A repository secret is available to any workflow job in the repository, on any branch that can run workflows.
Answer: The value would pass through the pipeline (logs, job memory, third-party actions) and be stored in the task definition, readable by anyone with ecs:DescribeTaskDefinition. Using secrets with valueFrom stores only an ARN; ECS fetches the value at runtime with the execution role.
Answer: It tells the runner to redact a value from all later log output. You need it for values that are secret but were not GitHub secrets — for example a token fetched or generated during the job — and you must call it before anything prints the value.
Answer: Build arguments are recorded in the image's build history and metadata, so anyone who can pull the image can read them. BuildKit secret mounts expose the secret to one RUN step only and leave nothing in the image.
Answer: Revoke or rotate the credential so the leaked value no longer works. Cleaning logs or Git history first leaves a valid credential in the attacker's hands.
Interview questions
- "How do you manage secrets in your pipelines?" — Eliminate cloud keys with OIDC; keep runtime secrets out of the pipeline (Secrets Manager referenced by ARN, read by the runtime role); scope the remaining build-time secrets to GitHub environments and prefer short-lived tokens; keep them out of images with BuildKit mounts and out of logs by never printing them.
- "What are the limits of GitHub secret masking?" — Exact-match only; transformed or structured values may not be masked; values that aren't registered secrets aren't masked unless you use
add-mask. Treat masking as a safety net. - "How do you rotate a database password without downtime?" — Use RDS-managed passwords or a Secrets Manager rotation strategy that keeps the previous version valid briefly; cache in the app with a short max age and refresh on authentication failure; rehearse in staging.
- "How would you prove to an auditor that the pipeline can't read customer data?" — Show the deploy role's policies and an IAM policy simulation denying
GetSecretValueon runtime secrets, the secret resource policy, and CloudTrail showing which principals read the secret.
Certification practice
An ASP.NET Core application runs on Amazon ECS with Fargate and needs a database password. The password must not be stored in the container image or in the CI/CD system and must support rotation. Which solution meets these requirements?
A. Store the password as a GitHub secret and pass it as an environment variable in the task definition.
B. Store the password in AWS Secrets Manager and reference the secret ARN in the task definition's secrets section; grant the task execution role permission to read it.
C. Bake the password into appsettings.Production.json in the image.
D. Store the password in an S3 object and download it at container start.
Answer: B. ECS injects the value at task start; the pipeline only handles the ARN; Secrets Manager supports rotation. A puts the secret in the pipeline and the task definition. C puts it in the image. D needs custom code and lacks rotation and fine-grained secret auditing.
A CodeBuild project needs an API token for a third-party scanning service. The token must not appear in the buildspec, the project configuration, or the build logs. What should the engineer do?
A. Add the token as a plaintext environment variable on the CodeBuild project.
B. Store the token in AWS Secrets Manager and reference it in the buildspec env secrets-manager section; grant the CodeBuild service role access to that secret only.
C. Commit the token to the repository in an encrypted file and decrypt it with a hard-coded key.
D. Pass the token on the command line in the build phase.
Answer: B. CodeBuild fetches it at build time using its service role; the buildspec holds only the reference. A stores the value in project configuration, visible to anyone who can describe the project. C just moves the secret to the key. D exposes it in process listings and possibly logs.
A security engineer must ensure that only the payments service's ECS task role can read a specific secret, even if an administrator in the same account attaches a broad IAM policy to another role. Which control achieves this?
A. A KMS key with automatic rotation.
B. A resource policy on the secret that denies secretsmanager:GetSecretValue unless aws:PrincipalArn is the task role.
C. Tag-based access control on the IAM roles.
D. Enabling CloudTrail data events.
Answer: B. An explicit deny in the resource policy overrides any IAM allow in the same account. A rotates key material but doesn't restrict callers. C depends on tag discipline and IAM policies that an admin could change. D records access but does not prevent it.
Completion checklist
- I inventoried the secrets in one pipeline and classified each one
- No stored AWS keys remain in my GitHub repositories
- My task definitions reference secrets by ARN, and my deploy role cannot read them
- My Dockerfiles use BuildKit secret mounts and
docker historyshows no secrets - I can explain the limits of masking and when to use
::add-mask:: - I deleted the lab secrets and scheduled the KMS key for deletion
Key takeaways
- Eliminate, isolate, protect: OIDC removes cloud keys; runtime secrets never pass through the pipeline; the few build-time secrets are scoped and short-lived.
- The pipeline deploys references to secrets (ARNs), not values.
- Masking is exact-match string replacement — design so secrets are never printed in any form.
- BuildKit secret mounts keep tokens out of image layers;
--build-argdoes not. - When a secret leaks: revoke first, then investigate, then clean up.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 9.2: an OIDC deploy role that a fork cannot assume
Evaluate real GitHub OIDC tokens against IAM trust policies offline, prove that a fork, a pull request, another repository and the wrong audience are all refused, lint dangerous policies, then create the provider and three least-privilege roles with Terraform.
Pipeline security: protect the machine that deploys your bank
Treat the delivery pipeline as a privileged, attackable system: stop script injection and untrusted pull requests, pin actions and base images, scope tokens and IAM roles, sign and attest artefacts, produce SBOMs, and audit workflows with a small C# linter plus zizmor.