Skip to Content
Cloud Engineering Academy
9. DevOps & CI/CD

Secrets in CI/CD: keep them out of pipelines, logs and images

Classify every secret a delivery pipeline touches, remove the ones OIDC makes unnecessary, keep runtime secrets out of the pipeline entirely, and handle the rest with GitHub environment secrets, AWS Secrets Manager, CodeBuild and BuildKit secret mounts so they never appear in logs or image layers.

intermediate⏱ 1 h 40 min🟢 FreeDOP-C02DVA-C03SCS-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SecretsManager 4.1.1.2) · Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3 · GitHub Actions (aws-actions/aws-secretsmanager-get-secrets v4.0.0, aws-actions/configure-aws-credentials v6.3.0) · Docker BuildKit · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Sort the secrets a pipeline touches into four kinds — cloud credentials, build-time secrets, deploy-time references and runtime secrets — and choose the right home for each.
  • Explain what GitHub repository, organisation and environment secrets protect, and where their masking stops working.
  • Fetch a secret from AWS Secrets Manager inside a GitHub Actions job through an OIDC role, and inside AWS CodeBuild through the buildspec.
  • Pass a private NuGet feed token to docker build with a BuildKit secret mount so it never lands in an image layer.
  • Deliver database credentials to an ASP.NET Core service on ECS without the pipeline ever seeing them, and survive a rotation with a small C# credential provider.
  • Respond to a leaked secret in the right order: revoke, rotate, investigate, then clean up history.

Prerequisites

Why this matters in banking

Three real patterns, each of which has caused public incidents across the industry, re-told with Harbourline names:

  1. The key in the pipeline. A team stored an IAM user's access key as a repository secret so the pipeline could deploy. A compromised third-party action in that workflow printed environment variables into the build log of a public repository. Within minutes, automated scanners had the key; within an hour, crypto-mining instances were running in the account. With OIDC there would have been no key to steal.
  2. The token in the image. A Dockerfile used ARG NUGET_TOKEN to restore packages from the bank's private feed. Build arguments are recorded in the image history. Anyone who could pull the image — including a vendor with read access to the registry — could read the token with docker history.
  3. The password in the log. A deployment script ran set -x for debugging and echoed a connection string built from a database password. GitHub masked the password, but the script had URL-encoded it first, and the encoded form was not masked.

For a bank these are not just hygiene issues. PCI DSS requires strong protection and restricted access for credentials to systems in scope; GDPR / UK GDPR breaches often start with a leaked credential; and auditors will ask how secrets are stored, who can read them, and how quickly they can be rotated. A good answer is short: "the pipeline holds no long-term credentials; runtime secrets never pass through it; the few build-time secrets are scoped to an environment, short-lived where possible, and kept out of logs and images by design."

The idea in plain language

A secret is any value that grants access: a password, an API token, a private key, a signing key, a connection string with a password in it. A delivery pipeline is dangerous for secrets because it runs lots of other people's code (actions, NuGet packages, build tools) and writes long, searchable logs.

The safest secret is one the pipeline never has. So the approach is to eliminate, then isolate, then protect:

KindExampleWhere it should liveDoes the pipeline see it?
Cloud credentialsAWS access keys for deployingNowhere — replaced by OIDC federation and short-lived role sessionsOnly a short-lived session, never a stored key
Build-time secretsPrivate NuGet feed token, SAST tool API tokenGitHub environment secret or Secrets Manager, fetched for one stepYes, briefly — so keep it scoped and out of logs and images
Deploy-time references"The DB password is in secret hl/accounts/db"In the IaC or task definition as an ARN, not a valueOnly the reference
Runtime secretsDatabase password, partner API key, card tokenisation keySecrets Manager (or KMS for keys), read by the running service's roleNo
Signing keysImage or artefact signing keyAWS KMS / AWS Signer, key never exportedNo — the pipeline asks the service to sign

Analogy: a cash-in-transit courier (the pipeline) carries sealed bags between branches. The courier never knows the vault combination (runtime secret): each branch's vault opens for that branch's staff only. The courier does carry a delivery pass (build-time secret), but it is valid for one route, one day, and it is never photocopied into the delivery log.

What problem the AWS services and tools solve

ToolWhat it gives youLimits and sharp edges
GitHub Actions secrets (repository, organisation, environment)Encrypted values injected into jobs; automatically masked in logs48 KB per secret; up to 100 repository and 100 environment secrets. Masking matches exact strings only — transformed values (base64, URL-encoded, split) are not masked. Avoid structured (JSON/YAML) secret values
Environment secretsAvailable only to jobs that reference the environment, and only after its protection rules (e.g. required reviewers) passAdministrators can bypass protection unless that is disabled for the environment
Fork protectionSecrets (other than GITHUB_TOKEN) are not passed to workflows triggered by pull requests from forkspull_request_target and workflow_run run with secrets in the base repository's context — dangerous with untrusted code
AWS Secrets ManagerEncrypted storage with KMS, fine-grained IAM, resource policies, versioning, automatic rotation, CloudTrail audit of every read$0.40 per secret per month plus API calls ⚠ VERIFY current pricing. Callers must cache to avoid cost and throttling
SSM Parameter Store SecureStringCheaper storage for configuration with some secret valuesNo built-in rotation; standard parameters are free, advanced ones are not
aws-actions/aws-secretsmanager-get-secretsFetches secrets into masked environment variables in a GitHub jobValues are only as safe as the steps that run after it
CodeBuild env.secrets-managerInjects secrets into the build container from Secrets ManagerThe CodeBuild service role needs secretsmanager:GetSecretValue on exactly those secrets
ECS task definition secretsThe ECS agent fetches the secret at task start and sets it as an environment variable in the containerRetrieved at start only; a rotated value is picked up when tasks are replaced
BuildKit --secretMounts a secret as a file (or env) for one RUN instruction only; nothing in layers or historyRequires BuildKit (default in current Docker)

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS / GitHub equivalentWhat's different
Azure DevOps secret variablesGitHub Actions secretsSimilar masking behaviour and similar blind spots for transformed values
Variable groups linked to Key Vaultaws-secretsmanager-get-secrets in a job, or CodeBuild env.secrets-managerAccess is through the job's OIDC role; scope per GitHub environment
Service connection with a client secretOIDC federation to an IAM roleNo secret to store or rotate
Key Vault references in App Service settingsECS task definition secrets with valueFrom an ARNThe platform resolves the secret at start; the deployment carries only the reference
Managed identity reading Key Vault at runtimeECS task role / Lambda execution role reading Secrets ManagerSame pattern; cache in process
Key Vault secret rotation with Event GridSecrets Manager rotation (Lambda-based, or managed for RDS master passwords)RDS can manage the master user password in Secrets Manager for you
Docker --build-arg secrets in Azure PipelinesBuildKit --secretBuild args persist in image metadata; secret mounts do not

Architecture

The important thing is what is missing: there is no arrow from Secrets Manager to the GitHub job. The pipeline deploys a task definition that names the secret; ECS fetches the value at runtime with the task execution role.

Security-boundary view

Step-by-step implementation

Inventory every secret the pipeline touches

Search the workflows for secrets. and for anything that looks like a credential in variables. For each, record: what it grants, who owns it, how it is rotated, and which of the four kinds it is. Most teams find at least one stored AWS key and one runtime secret that does not need to be there.

Eliminate cloud credentials

Replace every stored AWS access key with an OIDC role per environment. Delete the IAM user and its keys afterwards — leaving them "just in case" defeats the purpose.

Move runtime secrets out of the pipeline

Create the secret in Secrets Manager (or let RDS manage the master password). Reference its ARN in the ECS task definition secrets block or read it at runtime from Lambda. Remove the value from GitHub. Grant secretsmanager:GetSecretValue on that ARN to the task execution role (for injection) or task role (for runtime reads) — not to the deploy role.

Scope the build-time secrets that remain

Store them as environment secrets, not repository secrets, so they reach only the jobs that declare that environment. Prefer short-lived tokens: for example, AWS CodeArtifact authorisation tokens obtained with the OIDC role (valid for up to 12 hours) instead of a long-lived feed password.

Keep secrets out of images

Never use ARG or ENV for secrets in a Dockerfile. Use RUN --mount=type=secret so the secret exists only for that step. Use a multi-stage build so that tools and config files used during restore are not copied into the runtime image.

Keep secrets out of logs

Pass secrets to commands through environment variables, not command-line arguments. Never set -x in a step that handles secrets. If you derive a new value from a secret (an encoded form, a generated token), register it with ::add-mask:: before anything prints it. Don't upload workspace folders that may contain credentials files as artefacts.

Detect and respond

Turn on GitHub secret scanning and push protection where your plan allows it, and run Gitleaks in CI (see Pipeline security scanning). Write the leak runbook: revoke first, rotate, check CloudTrail for use of the leaked credential, then clean history.

AWS Console instructions

Create a secret:

  1. Console → Secrets Manager → Store a new secret.
  2. Other type of secret; add key/value pairs username and password (or let RDS manage the master password — see below).
  3. Encryption key: choose a customer managed KMS key (for example alias/hl-accounts-secrets) so you control the key policy.
  4. Name: hl/accounts/db. Add tags Project=cea-lab, Lab=m09-l11, Environment=learn.
  5. On the rotation page, enable rotation if you have a rotation function; otherwise leave it for now.

Let RDS manage the database master password:

  1. Console → RDS → Databases → Modify (or Create database).
  2. Under Credentials management, choose Managed in AWS Secrets Manager. RDS creates and rotates the secret; your application reads it like any other.

GitHub environment secret:

  1. GitHub → repository → Settings → Environments → prod → Environment secrets → Add secret.
  2. Name NUGET_FEED_TOKEN; paste the token. It is now available only to jobs with environment: prod, and only after approval.

Check who read a secret:

  1. Console → CloudTrail → Event history, filter Event name = GetSecretValue, and look at the userIdentity and requestParameters.secretId of each event.

AWS CLI

P=learn-sandbox

# 1. Create a secret encrypted with a customer managed key, with lab tags.
#    The value is read from a file so it never appears in shell history or process lists.
aws secretsmanager create-secret --name hl/accounts/db \
  --kms-key-id alias/hl-accounts-secrets \
  --secret-string file://db-secret.json \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m09-l11 Key=Environment,Value=learn \
  --profile $P
shred -u db-secret.json 2>/dev/null || rm -f db-secret.json

# 2. Restrict the secret with a resource policy: only the task execution role may read it.
aws secretsmanager put-resource-policy --secret-id hl/accounts/db \
  --resource-policy file://secret-policy.json --block-public-policy --profile $P

# 3. Validate the resource policy before attaching changes (Access Analyzer-backed check).
aws secretsmanager validate-resource-policy --secret-id hl/accounts/db \
  --resource-policy file://secret-policy.json --profile $P

# 4. See metadata only (never the value) — safe to run in pipelines.
aws secretsmanager describe-secret --secret-id hl/accounts/db \
  --query '{Name:Name,Rotation:RotationEnabled,LastRotated:LastRotatedDate,Kms:KmsKeyId}' --profile $P

# 5. Rotate now (requires a rotation function or an RDS-managed secret).
aws secretsmanager rotate-secret --secret-id hl/accounts/db --profile $P

# 6. Who has read this secret in the last day? (CloudTrail management events)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --start-time "$(date -u -d '-1 day' +%Y-%m-%dT%H:%M:%SZ)" \
  --query 'Events[].[EventTime,Username]' --output table --profile $P

# 7. A short-lived CodeArtifact token instead of a stored feed password
#    (run inside the job after configure-aws-credentials; valid up to 12 hours).
aws codeartifact get-authorization-token --domain harbourline --domain-owner 111122223333 \
  --duration-seconds 3600 --query authorizationToken --output text --profile $P

The resource policy (file secret-policy.json) denies everyone except the task execution role, even principals in the same account with broad IAM permissions:

secret-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OnlyAccountsApiExecutionRole",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "ArnNotEquals": {
          "aws:PrincipalArn": [
            "arn:aws:iam::111122223333:role/hl-accounts-api-exec",
            "arn:aws:iam::111122223333:role/hl-break-glass"
          ]
        }
      }
    }
  ]
}

Infrastructure as Code

taskdef.json (excerpt) — the pipeline deploys this; it contains no secret values
{
  "family": "hl-accounts-api",
  "executionRoleArn": "arn:aws:iam::111122223333:role/hl-accounts-api-exec",
  "taskRoleArn": "arn:aws:iam::111122223333:role/hl-accounts-api-task",
  "containerDefinitions": [
    {
      "name": "api",
      "image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/hl-accounts-api@sha256:REPLACED_BY_PIPELINE",
      "environment": [
        { "name": "Database__Host", "value": "hl-accounts.cluster-abc.us-east-1.rds.amazonaws.com" },
        { "name": "Database__Name", "value": "accounts" }
      ],
      "secrets": [
        { "name": "Database__Username",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:111122223333:secret:hl/accounts/db-AbCdEf:username::" },
        { "name": "Database__Password",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:111122223333:secret:hl/accounts/db-AbCdEf:password::" }
      ]
    }
  ]
}

The valueFrom format is secret-arn:json-key:version-stage:version-id; the two trailing colons mean "current version". The execution role needs secretsmanager:GetSecretValue on the secret and kms:Decrypt on its key.

Complete code

1. The GitHub Actions job: environment secret, Secrets Manager, masking

.github/workflows/build-image.yml
name: build-image
on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  build:
    runs-on: ubuntu-24.04
    environment: build          # environment secrets below exist only for this environment
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false   # do not leave GITHUB_TOKEN in .git/config for later steps

      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ vars.BUILD_ROLE_ARN }}
          aws-region: us-east-1

      # Fetch a third-party scanner token from Secrets Manager into a masked env var.
      - uses: aws-actions/aws-secretsmanager-get-secrets@e952bd7125c27e5be0060e50955b90a7e5addd99 # v4.0.0
        with:
          secret-ids: |
            SCANNER_TOKEN, hl/ci/scanner-token

      # A derived value must be masked explicitly BEFORE anything can print it.
      - name: Get a short-lived feed token
        run: |
          TOKEN=$(aws codeartifact get-authorization-token --domain harbourline \
            --domain-owner "${{ vars.ARTIFACT_ACCOUNT_ID }}" --duration-seconds 1800 \
            --query authorizationToken --output text)
          echo "::add-mask::$TOKEN"
          echo "FEED_TOKEN=$TOKEN" >> "$GITHUB_ENV"

      # The token reaches docker build as a BuildKit secret, not a build argument.
      - name: Build image
        run: |
          docker build \
            --secret id=nuget_token,env=FEED_TOKEN \
            -t hl-accounts-api:${GITHUB_SHA} .

      # Prove the token is not in the image history (fails the job if it is).
      - name: Check image history for the token
        run: |
          if docker history --no-trunc hl-accounts-api:${GITHUB_SHA} | grep -F -q "$FEED_TOKEN"; then
            echo "::error::Feed token found in image history"; exit 1
          fi

2. Dockerfile with a secret mount and a clean runtime stage

Dockerfile
# syntax=docker/dockerfile:1
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY nuget.config ./
COPY src/Harbourline.Accounts.Api/*.csproj src/Harbourline.Accounts.Api/
# The secret is mounted at /run/secrets/nuget_token for THIS step only.
# NuGet expands %NUGET_TOKEN% from nuget.config at restore time.
RUN --mount=type=secret,id=nuget_token \
    NUGET_TOKEN="$(cat /run/secrets/nuget_token)" \
    dotnet restore src/Harbourline.Accounts.Api/Harbourline.Accounts.Api.csproj
COPY . .
RUN dotnet publish src/Harbourline.Accounts.Api -c Release -o /app --no-restore

# The runtime stage copies only the published output: no nuget.config, no SDK, no token.
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
WORKDIR /app
COPY --from=build /app .
USER $APP_UID
ENTRYPOINT ["dotnet", "Harbourline.Accounts.Api.dll"]
nuget.config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
    <add key="harbourline" value="https://harbourline-111122223333.d.codeartifact.us-east-1.amazonaws.com/nuget/internal/v3/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <harbourline>
      <add key="Username" value="aws" />
      <!-- Expanded from the environment at restore time; the file holds no secret. -->
      <add key="ClearTextPassword" value="%NUGET_TOKEN%" />
    </harbourline>
  </packageSourceCredentials>
</configuration>

3. A rotation-aware database credential provider in C#

When the secret is injected by ECS, a rotation is only picked up when tasks restart. For services that must survive rotation without a restart, read the secret at runtime, cache it, and refresh once when the database rejects the password.

src/Harbourline.Accounts.Api/Data/DbCredentialProvider.cs
using System.Text.Json;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;

namespace Harbourline.Accounts.Api.Data;

public sealed record DbCredentials(string Username, string Password);

public interface IDbCredentialProvider
{
    Task<DbCredentials> GetAsync(CancellationToken ct);
    /// <summary>Forget the cached value, e.g. after the database rejected the password.</summary>
    void Invalidate();
}

public sealed class SecretsManagerDbCredentialProvider(
    IAmazonSecretsManager secrets, string secretId, TimeProvider clock, TimeSpan maxAge) : IDbCredentialProvider
{
    private readonly SemaphoreSlim _gate = new(1, 1);
    private DbCredentials? _cached;
    private DateTimeOffset _fetchedAt;

    public async Task<DbCredentials> GetAsync(CancellationToken ct)
    {
        if (_cached is { } fresh && clock.GetUtcNow() - _fetchedAt < maxAge) return fresh;

        await _gate.WaitAsync(ct);   // one refresh at a time, even under load
        try
        {
            if (_cached is { } c && clock.GetUtcNow() - _fetchedAt < maxAge) return c;

            var response = await secrets.GetSecretValueAsync(
                new GetSecretValueRequest { SecretId = secretId }, ct);

            using var doc = JsonDocument.Parse(response.SecretString);
            _cached = new DbCredentials(
                doc.RootElement.GetProperty("username").GetString()!,
                doc.RootElement.GetProperty("password").GetString()!);
            _fetchedAt = clock.GetUtcNow();
            return _cached;
        }
        finally
        {
            _gate.Release();
        }
    }

    public void Invalidate() => _cached = null;

    // Never let the password reach logs through ToString() or a debugger display.
    public override string ToString() => $"SecretsManagerDbCredentialProvider({secretId})";
}
src/Harbourline.Accounts.Api/Data/ResilientConnectionFactory.cs
using Npgsql;

namespace Harbourline.Accounts.Api.Data;

public sealed class ResilientConnectionFactory(IDbCredentialProvider credentials, string host, string database)
{
    // PostgreSQL SQLSTATE 28P01 = invalid_password.
    private const string InvalidPassword = "28P01";

    public async Task<NpgsqlConnection> OpenAsync(CancellationToken ct)
    {
        try
        {
            return await OpenOnceAsync(ct);
        }
        catch (PostgresException ex) when (ex.SqlState == InvalidPassword)
        {
            // The secret was probably rotated since we cached it: refresh once and retry.
            credentials.Invalidate();
            return await OpenOnceAsync(ct);
        }
    }

    private async Task<NpgsqlConnection> OpenOnceAsync(CancellationToken ct)
    {
        var c = await credentials.GetAsync(ct);
        var csb = new NpgsqlConnectionStringBuilder
        {
            Host = host,
            Database = database,
            Username = c.Username,
            Password = c.Password,
            SslMode = SslMode.VerifyFull       // TLS with certificate and host-name verification
        };
        var conn = new NpgsqlConnection(csb.ConnectionString);
        await conn.OpenAsync(ct);
        return conn;
    }
}
tests/Harbourline.Accounts.Api.Tests/DbCredentialProviderTests.cs
using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;
using Harbourline.Accounts.Api.Data;
using Microsoft.Extensions.Time.Testing;
using NSubstitute;
using Xunit;

public class DbCredentialProviderTests
{
    private static IAmazonSecretsManager SecretsReturning(params string[] passwords)
    {
        var sm = Substitute.For<IAmazonSecretsManager>();
        var responses = passwords
            .Select(p => new GetSecretValueResponse { SecretString = $$"""{"username":"accounts_app","password":"{{p}}"}""" })
            .ToArray();
        sm.GetSecretValueAsync(Arg.Any<GetSecretValueRequest>(), Arg.Any<CancellationToken>())
          .Returns(responses[0], responses.Skip(1).ToArray());
        return sm;
    }

    [Fact]
    public async Task Caches_within_max_age()
    {
        var sm = SecretsReturning("first", "second");
        var clock = new FakeTimeProvider();
        var provider = new SecretsManagerDbCredentialProvider(sm, "hl/accounts/db", clock, TimeSpan.FromMinutes(15));

        await provider.GetAsync(CancellationToken.None);
        clock.Advance(TimeSpan.FromMinutes(5));
        var second = await provider.GetAsync(CancellationToken.None);

        Assert.Equal("first", second.Password);
        await sm.Received(1).GetSecretValueAsync(Arg.Any<GetSecretValueRequest>(), Arg.Any<CancellationToken>());
    }

    [Fact]
    public async Task Invalidate_forces_a_fresh_read_after_rotation()
    {
        var sm = SecretsReturning("before-rotation", "after-rotation");
        var provider = new SecretsManagerDbCredentialProvider(sm, "hl/accounts/db", new FakeTimeProvider(), TimeSpan.FromHours(1));

        await provider.GetAsync(CancellationToken.None);
        provider.Invalidate();
        var refreshed = await provider.GetAsync(CancellationToken.None);

        Assert.Equal("after-rotation", refreshed.Password);
    }

    [Fact]
    public void ToString_never_contains_the_password()
    {
        var provider = new SecretsManagerDbCredentialProvider(
            SecretsReturning("s3cr3t"), "hl/accounts/db", new FakeTimeProvider(), TimeSpan.FromMinutes(1));
        Assert.DoesNotContain("s3cr3t", provider.ToString());
    }
}

The test project references xunit.v3 4.0.3, NSubstitute and Microsoft.Extensions.TimeProvider.Testing (for FakeTimeProvider). Register the provider in Program.cs with builder.Services.AddAWSService<IAmazonSecretsManager>() and a singleton SecretsManagerDbCredentialProvider, so all requests share one cache.

Code explained line by line

Workflow

  1. environment: build — environment secrets and the BUILD_ROLE_ARN variable are only available to this job.
  2. persist-credentials: false — actions/checkout otherwise stores the GITHUB_TOKEN in the local Git config, where every later step (including third-party actions) can read it.
  3. aws-secretsmanager-get-secrets with SCANNER_TOKEN, hl/ci/scanner-token — fetches the secret through the job's OIDC role and exposes it as a masked environment variable named SCANNER_TOKEN.
  4. echo "::add-mask::$TOKEN" — the CodeArtifact token was not a GitHub secret, so GitHub does not know to hide it. Masking must happen before any step could print it.
  5. echo "FEED_TOKEN=$TOKEN" >> "$GITHUB_ENV" — passes the value to later steps as an environment variable, not as a step output, which would be visible to other jobs.
  6. --secret id=nuget_token,env=FEED_TOKEN — BuildKit reads the value from the environment variable and offers it to the build as a secret mount.
  7. The history check is a cheap regression test: if anyone reintroduces ARG NUGET_TOKEN, the job fails.

Dockerfile

  1. # syntax=docker/dockerfile:1 — selects a modern Dockerfile frontend that supports secret mounts.
  2. RUN --mount=type=secret,id=nuget_token — the file /run/secrets/nuget_token exists only while this RUN executes; it is not in any layer.
  3. NUGET_TOKEN="$(cat ...)" dotnet restore — sets the variable for that one command only.
  4. The runtime stage copies /app only; nuget.config, the SDK and any restore caches stay in the discarded build stage.
  5. USER $APP_UID — the official .NET images define a non-root user; run as it.

SecretsManagerDbCredentialProvider

  1. Primary-constructor parameters (secrets, secretId, clock, maxAge) — TimeProvider makes time testable.
  2. Fast path returns the cached value without locking when it is fresh enough.
  3. SemaphoreSlim — when the cache expires under load, only one request calls Secrets Manager; the others wait and reuse the result (avoids a burst of API calls and throttling).
  4. Double-check inside the lock — another caller may have refreshed while this one waited.
  5. JsonDocument.Parse(response.SecretString) — RDS-managed and most manual secrets store username and password as JSON keys.
  6. Invalidate() — called when the database says the password is wrong.
  7. ToString() override — prevents accidental logging of internals.

ResilientConnectionFactory

  1. SqlState == "28P01" — PostgreSQL's code for an invalid password, the signal that rotation happened.
  2. Retry once: a second failure is a real problem and must surface, not loop.
  3. SslMode.VerifyFull — encrypt in transit and verify the server's certificate and host name; credentials should never cross the network in clear.

Security considerations

⚠ Learning shortcut — not for production

In this lesson the secret's resource policy names a hl-break-glass role so you can still read it while experimenting. In production, the break-glass path should require MFA through IAM Identity Center, raise an alert on every use, and be reviewed afterwards — reading a production database password is a significant event.

⚠ Learning shortcut — not for production

The lab secret uses a 15-minute in-process cache and no automatic rotation. In production, enable rotation (or RDS-managed passwords), choose a cache age shorter than the rotation's overlap window, and test a rotation in staging before relying on it.

  • Masking is a safety net, not a control. It only matches exact strings. Base64, URL-encoding, JSON-escaping, splitting across lines or printing character by character all bypass it. Design so secrets are never printed in any form.
  • Structured secrets mask badly. A JSON secret stored in GitHub may not be redacted correctly; store one value per secret.
  • pull_request_target and workflow_run run with the base repository's secrets. Never check out and build untrusted pull-request code in those workflows.
  • Self-hosted runners keep state between jobs unless they are ephemeral. A secret written to disk by one job can be read by the next. Use ephemeral runners (or GitHub-hosted runners) for anything that handles secrets.
  • Terraform state and CloudFormation templates can contain secret values. Prefer generated or managed secrets; never put a secret in a CloudFormation parameter without NoEcho, and remember NoEcho does not hide values that you reference elsewhere in outputs or metadata.
  • Third-party actions see your environment. Every action in a job that has a secret can read it. Pin actions by commit SHA and keep the secret in as few steps (ideally jobs) as possible — see Pipeline security.

Banking use case

Harbourline's secrets inventory for the Payments Service pipeline, before and after this lesson's changes:

SecretBeforeAfter
AWS deploy credentialsIAM user key in repository secretRemoved — OIDC role per environment
Database passwordRepository secret, passed to aws ecs register-task-definition as an environment valueSecrets Manager (RDS-managed, rotated); task definition holds only the ARN
Sanctions-screening vendor API keyRepository secret, read by integration tests and by the appTwo keys: a sandbox key as a test environment secret for tests; the production key in Secrets Manager, read only by the running service
Private NuGet feed passwordOrganisation secret, passed as --build-argCodeArtifact token from the OIDC role, BuildKit secret mount
SAST tool tokenRepository secretbuild environment secret
Image signing keyPrivate key file in a repository secretAWS Signer / ECR managed signing — no key leaves AWS

The before-state had six long-lived secrets readable by any workflow on any branch. The after-state has two scoped, short-lived or managed secrets in GitHub; nothing in GitHub can read customer data.

Common mistakes

  • Passing secrets as command-line arguments → visible in process listings and sometimes in logs → pass by environment variable or file.
  • ARG / ENV for secrets in Dockerfiles → stored in image metadata forever → BuildKit secret mounts.
  • Debugging with set -x → every expanded variable is printed, encoded forms are not masked → never in steps that touch secrets.
  • Uploading the whole workspace as an artefact → credentials files and .env files travel with it → upload explicit paths only.
  • Deploy role can read runtime secrets → a compromised pipeline can read the database password → only runtime roles get GetSecretValue.
  • Rotating without testing the application → outage at the first automatic rotation → rehearse rotation in staging; implement refresh-on-auth-failure.
  • Cleaning Git history before revoking a leaked secret → the secret is still valid and still in forks, caches and clones → revoke first, always.

Troubleshooting

SymptomLikely causeFix
ECS task stops with ResourceInitializationError: unable to pull secretsExecution role lacks GetSecretValue or kms:Decrypt, or no network path to Secrets Manager from private subnetsAdd permissions; add a VPC endpoint for Secrets Manager (or NAT)
Secret appears in the log in an encoded formMasking only matches exact valuesMask the derived value with ::add-mask:: first; better, stop printing it
AccessDeniedException reading a secret despite an allow in IAMExplicit deny in the secret's resource policy, or KMS key policy does not allow the roleCheck the resource policy and the key policy
Secrets empty in a pull-request workflowPR is from a fork; secrets are withheld by designRun secret-dependent jobs only after merge, or with approval for first-time contributors; do not switch to pull_request_target
App fails after rotation with 28P01Cached old password; no refresh pathAdd invalidate-and-retry; replace tasks if using ECS-injected secrets
docker build error secret nuget_token not foundBuild run without --secret or BuildKit disabledPass --secret id=...; use a current Docker with BuildKit
Environment secret not available in a jobJob does not declare environment:, or approval pendingAdd environment: <name>; check the run's pending deployments

Cost and cleanup

Cost: GitHub secrets are free. A Secrets Manager secret costs about $0.40 per month (pro-rated) plus $0.05 per 10,000 API calls ⚠ VERIFY current pricing; new secrets may have a free trial period. A customer managed KMS key costs about $1 per month (pro-rated) plus requests. If you keep a secret and a key for one day and delete them, the cost is a few cents. RDS (Terraform tab) is not needed for this lesson — skip it, or deploy and destroy within an hour. Label: free (within cents).

P=learn-sandbox
# Delete the secret immediately (no recovery window) — lab only.
aws secretsmanager delete-secret --secret-id hl/accounts/db --force-delete-without-recovery --profile $P
aws secretsmanager delete-secret --secret-id hl/ci/scanner-token --force-delete-without-recovery --profile $P

# Schedule the KMS key for deletion (minimum waiting period is 7 days; billing stops when deleted).
KEY=$(aws kms describe-key --key-id alias/hl-accounts-secrets --query KeyMetadata.KeyId --output text --profile $P)
aws kms delete-alias --alias-name alias/hl-accounts-secrets --profile $P
aws kms schedule-key-deletion --key-id "$KEY" --pending-window-in-days 7 --profile $P

# Confirm nothing tagged for this lesson remains (scheduled-for-deletion keys may still show).
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m09-l11 \
  --query 'ResourceTagMappingList[].ResourceARN' --profile $P

Exercises

  • 🟢 Guided: Build the Dockerfile above twice: once with ARG NUGET_TOKEN and --build-arg, once with the secret mount. Run docker history --no-trunc on both and find the token in the first. Then create a secret in Secrets Manager and fetch it in a GitHub job with aws-secretsmanager-get-secrets; try echo "$SCANNER_TOKEN" and echo "$SCANNER_TOKEN" | base64 and compare what the log shows.
  • 🟡 Challenge: Write an ECS task definition and IAM policies for the Accounts API where the deploy role can register the task definition and pass the roles, but cannot read the database secret. Prove it with aws iam simulate-principal-policy against both roles.
  • 🔴 Live problem: At 14:05 a developer notices that a sanctions-vendor API key appears in a public GitHub Actions log from yesterday. The key is used by the production Payments Service. Write the incident timeline for the next two hours: who you call, what you revoke and in which order, how you keep payments flowing, what you check in logs, and what you change so it cannot recur.

Assignment

Add a SECRETS.md to your portfolio repository with: an inventory table (secret, kind, store, scope, rotation, owner); a diagram showing which roles can read which secrets; the leak runbook; and evidence (screenshots or command output) that your image history contains no secrets and that your deploy role cannot read runtime secrets.

Solutions

Knowledge check

Interview questions

  • "How do you manage secrets in your pipelines?" — Eliminate cloud keys with OIDC; keep runtime secrets out of the pipeline (Secrets Manager referenced by ARN, read by the runtime role); scope the remaining build-time secrets to GitHub environments and prefer short-lived tokens; keep them out of images with BuildKit mounts and out of logs by never printing them.
  • "What are the limits of GitHub secret masking?" — Exact-match only; transformed or structured values may not be masked; values that aren't registered secrets aren't masked unless you use add-mask. Treat masking as a safety net.
  • "How do you rotate a database password without downtime?" — Use RDS-managed passwords or a Secrets Manager rotation strategy that keeps the previous version valid briefly; cache in the app with a short max age and refresh on authentication failure; rehearse in staging.
  • "How would you prove to an auditor that the pipeline can't read customer data?" — Show the deploy role's policies and an IAM policy simulation denying GetSecretValue on runtime secrets, the secret resource policy, and CloudTrail showing which principals read the secret.

Certification practice

Completion checklist

  • I inventoried the secrets in one pipeline and classified each one
  • No stored AWS keys remain in my GitHub repositories
  • My task definitions reference secrets by ARN, and my deploy role cannot read them
  • My Dockerfiles use BuildKit secret mounts and docker history shows no secrets
  • I can explain the limits of masking and when to use ::add-mask::
  • I deleted the lab secrets and scheduled the KMS key for deletion

Key takeaways

  • Eliminate, isolate, protect: OIDC removes cloud keys; runtime secrets never pass through the pipeline; the few build-time secrets are scoped and short-lived.
  • The pipeline deploys references to secrets (ARNs), not values.
  • Masking is exact-match string replacement — design so secrets are never printed in any form.
  • BuildKit secret mounts keep tokens out of image layers; --build-arg does not.
  • When a secret leaks: revoke first, then investigate, then clean up.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page