Docker and Amazon ECR in pipelines
Build a small, reproducible, non-root .NET 10 container image, tag it so every running container can be traced to a commit, push it to Amazon ECR through a narrowly scoped role, scan it before and after the push, and deploy it by digest.
Assumes: .NET 10 LTS · mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled · Docker BuildKit · Amazon ECR · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what makes a container build reproducible and which parts are never fully reproducible without extra effort.
- Read the lab's multi-stage
Dockerfileline by line and say why each instruction is there. - Choose between tags and digests, apply a tagging policy (commit tag plus semantic version, never
latest) and enforce it with ECR tag immutability. - Create an ECR repository with scan on push, encryption and a lifecycle policy, using CDK (C#) or Terraform.
- Give a pipeline role the smallest set of ECR actions needed to push, and explain why
ecr:GetAuthorizationTokenis the one action that cannot be limited to a repository. - Build locally first, scan, then push; attach provenance and an SBOM to the image; and hand the digest to the deploy stage.
Prerequisites
- GitHub Actions with OIDC — how the pipeline obtains a role without stored keys.
- AWS CodeBuild, CodePipeline and CodeDeploy — the build side if you use AWS-native tooling.
- Module 3: .NET on AWS — containers, ECR and ECS basics, and the Module 3 lab on pushing to ECR.
- Pipeline security scanning — what each scanner finds.
- Tools: Docker Desktop with BuildKit (the default), AWS CLI v2 with the
learn-sandboxprofile, .NET 10 SDK or Docker.
Why this matters in banking
A container image is the unit you ship. Three questions come up in every change-board meeting about it:
- "What is running in production right now?" If the answer is "the image tagged
latest", nobody can say, becauselatestis a label that moves. A bank needs a statement like "commit3f9c2ab1d4e5, built by run 1234567890, digestsha256:4b1f...". - "Is it the same image that passed testing?" If the pipeline rebuilds for production, the bytes differ, and the test evidence no longer applies. The image must be built once and promoted.
- "What is inside it, and does it contain known vulnerabilities or a shell an attacker can use?" A large base image with a package manager, a shell and curl gives an intruder tools; a minimal image gives them very little.
A registry misconfiguration also matters: a repository that allows tags to be overwritten, or a push role that can write to every repository in the account, lets one compromised pipeline replace another team's production image.
The idea in plain language
An image is a layered, read-only package of your application and its runtime. A registry stores images. Amazon ECR (Elastic Container Registry) is AWS's private registry. A pipeline does four things with it:
- Build the image from a
Dockerfile. - Tag it, giving it human-readable names.
- Push it to ECR, which stores each layer and a manifest; the SHA-256 hash of the manifest is the image's digest.
- Deploy by telling ECS (or Lambda, or Kubernetes) which image to run.
Analogy: a tamper-evident evidence bag. The tag is the label someone wrote on the bag, which anyone can rewrite. The digest is the numbered security seal: if one byte inside changes, the seal number changes. A careful bank quotes the seal number, not the label.
What problem the AWS service solves
| Need | How ECR helps | Limits and sharp edges |
|---|---|---|
| Private storage close to compute | Per-account, per-Region registry; pulls from ECS and Fargate over AWS networks | Pulls from other Regions cost data transfer; use replication or a regional repository |
| Access control | IAM identity policies and repository policies; separate push and pull permissions | ecr:GetAuthorizationToken is account-wide (no resource-level restriction) |
| Tag safety | Tag immutability: a tag cannot be re-pointed at different content | Immutable tags mean a failed release version cannot be re-pushed; bump the version |
| Vulnerability scanning | Basic scanning on push (and manual); enhanced scanning with Amazon Inspector, continuous | Enhanced scanning is a paid Inspector feature ⚠ VERIFY pricing; scanning finds known CVEs, not logic flaws or malware |
| Cost control | Lifecycle policies expire old and untagged images | Policies are evaluated on a schedule; they do not delete instantly |
| Encryption | AES-256 by default; optional AWS KMS key | A customer managed key adds a KMS charge and a key policy to maintain |
| Distribution | Cross-Region and cross-account replication; pull-through cache for public registries | Replication is asynchronous; pull-through cache must be allowed by policy |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Container Registry (ACR) | Amazon ECR | One registry per account and Region; repositories are created individually (no wildcard push to a new name unless you allow creation on push or use repository creation templates ⚠ VERIFY) |
az acr login | aws ecr get-login-password piped to docker login | The password is a token valid for 12 hours, obtained with the caller's IAM credentials |
| ACR tasks | CodeBuild or GitHub Actions | ECR only stores; building is a separate service |
| ACR repository-scoped tokens and RBAC roles (AcrPush, AcrPull) | IAM policies and repository policies on specific actions | AWS has no "AcrPush" role; you assemble the action list |
| Defender for Containers scanning | ECR basic scanning, or enhanced scanning with Amazon Inspector | Basic scanning is included; enhanced is billed through Inspector |
| ACR content trust / image signing | Sigstore Cosign, Notation with AWS Signer, or build provenance attestations | Signing is a separate step you add |
mcr.microsoft.com/dotnet/aspnet base images | The same images; pull-through cache can mirror them | You can keep a copy in your own registry so builds do not depend on the public one |
Architecture
Security-boundary view
The push role cannot touch other repositories. The runtime role can only pull. Neither can delete images or change repository policy.
Reproducible builds
A reproducible build produces the same output from the same inputs. Perfect bit-for-bit reproducibility is hard for containers (timestamps, file ordering, base-image updates), so aim for repeatable and traceable:
| Source of drift | What causes it | What you do |
|---|---|---|
| Base image changes | FROM ...:10.0 is a moving tag; Microsoft publishes patched images | Pin by digest in production builds and let Dependabot raise reviewed updates; accept that unpinned builds pick up patches |
| Package versions | NuGet floating versions or missing lock files | Pin versions; use RestorePackagesWithLockFile and --locked-mode in CI ⚠ VERIFY the flag name for your SDK |
| Build inputs | Local files that are not in Git | Build only from a clean checkout in CI; use .dockerignore |
| Timestamps and paths | Compiler embeds paths | ContinuousIntegrationBuild=true in the publish step (the lab sets it) normalises source paths |
| Different tools per machine | Developer SDK differs from CI | global.json pins the SDK version; build in the SDK container |
| Hidden network fetches | curl in a build step | Avoid; if needed, verify checksums |
The lab Dockerfile, line by line
Open labs/m09-delivery-pipeline/Dockerfile. It is a two-stage build.
# syntax=docker/dockerfile:1
ARG VERSION=0.1.0
ARG GIT_SHA=local
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:10.0 AS build
ARG TARGETARCH
ARG VERSION
ARG GIT_SHA
WORKDIR /source
# 1) Restore using only the project files, so this layer is cached until dependencies change.
COPY global.json Directory.Build.props ./
COPY src/Harbourline.Accounts.Api/Harbourline.Accounts.Api.csproj src/Harbourline.Accounts.Api/
RUN dotnet restore src/Harbourline.Accounts.Api/Harbourline.Accounts.Api.csproj -a $TARGETARCH
# 2) Copy the source and publish.
COPY src/Harbourline.Accounts.Api/ src/Harbourline.Accounts.Api/
RUN dotnet publish src/Harbourline.Accounts.Api/Harbourline.Accounts.Api.csproj \
-c Release -a $TARGETARCH --no-restore -o /app \
-p:UseAppHost=false -p:Version=$VERSION -p:SourceRevisionId=$GIT_SHA -p:ContinuousIntegrationBuild=true
FROM mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled AS final
ARG VERSION
ARG GIT_SHA
WORKDIR /app
COPY --from=build /app .
LABEL org.opencontainers.image.title="harbourline-accounts-api" \
org.opencontainers.image.version="$VERSION" \
org.opencontainers.image.revision="$GIT_SHA" \
org.opencontainers.image.vendor="Harbourline Bank (fictional)"
USER $APP_UID
ENV ASPNETCORE_HTTP_PORTS=8080 \
DOTNET_gcServer=0
EXPOSE 8080
ENTRYPOINT ["dotnet", "Harbourline.Accounts.Api.dll"]# syntax=docker/dockerfile:1— selects the current stable Dockerfile frontend, so newer features such as secret mounts behave consistently across Docker versions.ARG VERSIONandARG GIT_SHA— build-time inputs for version and commit. They are not secrets; build arguments are recorded in image history, which is exactly why the secrets lesson forbids using them for credentials.FROM --platform=$BUILDPLATFORM ... AS build— the build stage runs on the builder's native architecture (fast), whileTARGETARCHtellsdotnetwhich architecture to publish for. This is how an amd64 CI runner produces an arm64 image for Graviton without emulation. ⚠ VERIFY that your base image tag has a manifest for both architectures.COPY global.json Directory.Build.props ./and the project-file-onlyCOPY— copies the smallest set of files that determine dependencies. Docker caches theRUN dotnet restorelayer until one of those files changes, so ordinary code changes reuse the restored packages. Order instructions from least to most frequently changing.RUN dotnet restore ... -a $TARGETARCH— restore for the target architecture. With NuGet Audit enabled inDirectory.Build.props, restore fails on high or critical advisories, which is a supply-chain gate at build time.COPY src/...thenRUN dotnet publish ... --no-restore— publishes a framework-dependent release build.--no-restoreguarantees publish does not fetch anything new.UseAppHost=falsedrops the native launcher you do not need.VersionandSourceRevisionIdstamp the assembly so the API's/versionendpoint can report the commit.ContinuousIntegrationBuild=truemakes the compiler produce deterministic paths.FROM mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled AS final— the runtime stage. Chiseled images are Ubuntu-based, contain only what .NET needs, have no shell and no package manager, and run as a non-root user. Anything that needsshin the container (debugging withdocker exec, shell-based health checks) will not work, which is a deliberate trade-off.COPY --from=build /app .— only the publish output crosses into the final image. The SDK, source code and NuGet cache stay behind.LABEL org.opencontainers.image.*— standard OCI labels. Scanners, ECR and auditors can read version and revision from the image itself.USER $APP_UID— the chiseled image defines a built-in non-root user ID; this line makes it explicit. The ECS task definition addsreadonlyRootFilesystemand drops Linux capabilities.ENV ASPNETCORE_HTTP_PORTS=8080— listen on an unprivileged port. A non-root process cannot bind port 80.DOTNET_gcServer=0— workstation garbage collection is a better default for small containers than server GC.- No
HEALTHCHECK— a chiseled image has nocurl. The load balancer probes/health/readyinstead, and the lab's blue/green configuration uses that. ENTRYPOINT ["dotnet", "Harbourline.Accounts.Api.dll"]— exec form, so the process is PID 1 and receives termination signals, allowing graceful shutdown during deployments.
And .dockerignore keeps bin/, obj/, .git and test projects out of the build context, so the context is small and secrets in local files are not sent to the daemon.
Tags and digests
Tag (1.4.0, git-3f9c2ab1d4e5) | Digest (@sha256:4b1f...) | |
|---|---|---|
| What it is | A movable label | A hash of the image manifest |
| Can it change? | Yes, unless the repository is immutable | Never |
| Who reads it | Humans, release notes | Machines, evidence, deploy stages |
| Use in deployment | Convenient to choose a version | Reference this in task definitions |
The lab's policy (implemented in src/Harbourline.Delivery/Images/ImageTagPolicy.cs and used by both workflows):
- Every build gets
git-<first 12 characters of the commit SHA>: traceability. - Releases also get the semantic version, for example
1.4.0: what people talk about. - No
latest,stable,prod,productionorcurrent: moving names make it impossible to say what runs. - Deployments reference the digest:
repo@sha256:....
Try the tool from the lab root:
dotnet run --project src/Harbourline.Delivery -- image-tags 3f9c2ab1d4e5f60718293a4b5c6d7e8f90a1b2c3 1.4.0
# git-3f9c2ab1d4e5
# 1.4.0
dotnet run --project src/Harbourline.Delivery -- image-tags 3f9c2ab1 1.4.0 # exit 1: use the full 40-character SHAImmutable tags
With image_tag_mutability = "IMMUTABLE", ECR refuses a push that would move an existing tag to different content. Pushing the same tag with the same content is accepted ⚠ VERIFY the behaviour for an identical manifest. That is the registry-side enforcement of your tagging policy. The consequence for pipelines: a re-run of a release job that tries to push the same version fails with tag invalid: The image tag '1.4.0' already exists, which is correct. The fix is a new version, not an overwrite. AWS has been extending immutability with exclusion filters for specific tag patterns; if you need mutable tags for something like a cache tag, check the current options ⚠ VERIFY before you relax the setting for a whole repository.
Scanning: before and after the push
| Stage | Tool | What it catches | Gate? |
|---|---|---|---|
| Restore | NuGet Audit | Known advisories in .NET dependencies | Fail on high and critical |
| Candidate image (before push) | Trivy (or another scanner) | OS and library CVEs in the image | Fail on fixable critical findings |
| On push | ECR basic scanning | CVEs from a standard database | Report; check findings before deploy |
| Continuous | ECR enhanced scanning with Amazon Inspector | New CVEs in images already stored, and language packages | Alert; opens a patch workflow |
| Weekly | Scheduled rebuild and scan | New CVEs against unchanged code | Alert |
Scan the candidate before pushing so a vulnerable image never reaches the registry where someone might deploy it by mistake. The lab's release.yml builds locally with load: true, push: false, runs Trivy with severity: CRITICAL, ignore-unfixed: true and exit-code: "1", and only then pushes. The action is pinned to a full commit SHA, version v0.36.0; the lab notes that tags older than v0.35.0 should never be used because of a March 2026 supply-chain compromise of the Trivy action ⚠ VERIFY the incident details before you cite it. Details of tool choice and thresholds are in Pipeline security scanning.
Why ignore-unfixed? A CVE with no available fix cannot be remediated by rebuilding, and failing every build for it trains people to ignore the scanner. Record it as a reviewed, time-limited exception instead.
Pushing with a narrow role
The push role (hl-m09-gh-ecr-push, trusted only for the main branch via OIDC) has exactly two statements:
{
"Version": "2012-10-17",
"Statement": [
{ "Sid": "LogInToEcr", "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" },
{
"Sid": "PushToOneRepository",
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability", "ecr:InitiateLayerUpload", "ecr:UploadLayerPart",
"ecr:CompleteLayerUpload", "ecr:PutImage", "ecr:BatchGetImage",
"ecr:DescribeImages", "ecr:DescribeImageScanFindings"
],
"Resource": "arn:aws:ecr:us-east-1:111122223333:repository/hl-m09-accounts-api"
}
]
}ecr:GetAuthorizationTokenhas no resource-level scope. It returns a registry login token for the account, valid for 12 hours; what the token can do is still limited by the other statements. KeepResource: "*"here and nowhere else.BatchCheckLayerAvailability,InitiateLayerUpload,UploadLayerPart,CompleteLayerUpload,PutImageare the push actions.BatchGetImageis included because build tooling may read the manifest it has just pushed (for example when attaching attestations or looking up the digest).DescribeImagesandDescribeImageScanFindingslet the job read the digest and the scan result for evidence.- Not included:
ecr:DeleteRepository,ecr:BatchDeleteImage,ecr:SetRepositoryPolicy,ecr:PutLifecyclePolicy,ecr:CreateRepository. A pipeline that can delete images or change repository policy can erase evidence. Repository settings are managed by IaC through a separate, reviewed path.
The pull side is a different principal: the ECS task execution role gets ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer and ecr:BatchGetImage on the repository, plus CloudWatch Logs actions for its log group. The AWS managed policy for task execution is broader than needed in some respects, so many banks write a tighter one ⚠ VERIFY the managed policy's current contents.
BuildKit cache, provenance and SBOM
- Layer cache in GitHub Actions:
docker/build-push-actioncan use the GitHub Actions cache backend (cache-from: type=gha,cache-to: type=gha,mode=max). Cache speeds builds but is a place to hide bad inputs; the final release job should be able to run with--no-cacheand produce the same result. See the Docker cache documentation under Further reading. - Provenance attestation:
provenance: mode=maxasks BuildKit to attach an attestation describing how the image was built (source repository, commit, builder, build arguments). In the lab it is stored alongside the image in the registry as an OCI artefact. - SBOM:
sbom: truegenerates a software bill of materials for the image and attaches it; the lab also generates an SPDX JSON file withanchore/sbom-actionas a build artefact for the evidence pack. The SBOM lets you answer "does any running image contain library X?" in minutes when a new CVE appears. - GitHub build-provenance attestation:
actions/attest-build-provenancesigns a statement tying the image digest to the workflow run, using GitHub's Sigstore-based attestation service ⚠ VERIFY whether attestations are available for your repository visibility and plan.
Because attestations are stored as additional manifests in the repository, an ECR lifecycle policy that expires "untagged" images may delete them. Check that your lifecycle rules keep attestation manifests that you want to retain ⚠ VERIFY how ECR treats referring artefacts in lifecycle rules.
Step-by-step implementation
Create the repository with the guardrails on
Immutable tags, scan on push, encryption, a lifecycle policy, and tags Project, Lab, Environment, Owner. Do it with IaC (below) so the settings are reviewed and reproducible.
Build locally and run the image
Build with version and commit stamped in, run it, and check /health/ready and /version. Run it with -e Health__ForceUnready=true to see the readiness endpoint return 503 (this is what you will use for the rollback exercise in the ECS lesson).
Compute and validate the tags
Use the lab tool so the pipeline and a developer's laptop agree.
Scan the candidate image
Fail on fixable critical findings. Save the JSON report as evidence.
Push with the narrow role
The job assumes ecr-push through OIDC, logs in to ECR, pushes both tags, and captures the digest.
Hand the digest on
Output repo@sha256:... from the build job. The test and prod deploy jobs render the task definition with exactly that reference.
Prove the guardrails
Try to push the same version tag twice (should fail), try to push to a different repository with the push role (should fail), and read the scan findings.
AWS Console instructions
Create a repository: Console → Amazon ECR → Private registry → Repositories → Create repository. Name hl-m09-accounts-api. Tag immutability: Enabled. Encryption: AES-256 (or AWS KMS with a customer managed key for stricter control). Scan on push is a repository or registry setting; enable it. Add tags.
Lifecycle policy: open the repository → Lifecycle policy → Create rule. Rule 1 priority 1: Untagged images, Since image pushed 3 days, action Expire. Rule 2 priority 2: Any image, Image count more than 15, action Expire.
Read a scan: repository → select an image tag → Scan results. Basic scanning lists CVE identifiers with severities. For enhanced scanning, open Amazon Inspector for the finding history.
Registry settings: Amazon ECR → Private registry → Settings hold the scanning type (basic or enhanced), replication rules and pull-through cache rules ⚠ VERIFY the current console layout.
AWS CLI
P=learn-sandbox
REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $P)
REGISTRY="$ACCOUNT.dkr.ecr.$REGION.amazonaws.com"
REPO=hl-m09-accounts-api
# 1. Create the repository with the guardrails on (immutable tags, scan on push).
aws ecr create-repository --repository-name $REPO \
--image-tag-mutability IMMUTABLE \
--image-scanning-configuration scanOnPush=true \
--encryption-configuration encryptionType=AES256 \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m09-delivery-pipeline Key=Environment,Value=learn \
--profile $P --region $REGION
# 2. Log in. The password is a 12-hour token obtained with your current IAM credentials.
aws ecr get-login-password --region $REGION --profile $P | docker login --username AWS --password-stdin $REGISTRY
# 3. Build with version and commit stamped in, tag with the commit tag and the version.
SHA=$(git rev-parse HEAD)
TAG="git-${SHA:0:12}"
docker build --build-arg VERSION=1.4.0 --build-arg GIT_SHA=$SHA \
-t $REGISTRY/$REPO:$TAG -t $REGISTRY/$REPO:1.4.0 .
# 4. Push both tags.
docker push $REGISTRY/$REPO:$TAG
docker push $REGISTRY/$REPO:1.4.0
# 5. Look up the digest. This reference is what you deploy.
DIGEST=$(aws ecr describe-images --repository-name $REPO --image-ids imageTag=$TAG \
--query 'imageDetails[0].imageDigest' --output text --profile $P --region $REGION)
echo "$REGISTRY/$REPO@$DIGEST"
# 6. Prove immutability: pushing the version tag with different content must fail.
echo "# touch" >> README.md && docker build -t $REGISTRY/$REPO:1.4.0 . && docker push $REGISTRY/$REPO:1.4.0
# Expected: tag invalid: The image tag '1.4.0' already exists in the repository and cannot be overwritten
# 7. Read the scan result for the image.
aws ecr describe-image-scan-findings --repository-name $REPO --image-id imageTag=$TAG \
--query 'imageScanFindings.findingSeverityCounts' --profile $P --region $REGION
# 8. Preview what a lifecycle policy WOULD delete before you apply it.
aws ecr start-lifecycle-policy-preview --repository-name $REPO --profile $P --region $REGION
aws ecr get-lifecycle-policy-preview --repository-name $REPO --profile $P --region $REGIONCommand 6 changes a file in your working copy; use a scratch clone, or skip the README edit and use a different --build-arg VERSION to get different content.
Infrastructure as code
using Amazon.CDK;
using Amazon.CDK.AWS.ECR;
using Amazon.CDK.AWS.IAM;
using Constructs;
namespace Harbourline.Delivery.Infra;
public sealed class EcrStack : Stack
{
public Repository Repo { get; }
public EcrStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
Repo = new Repository(this, "AccountsApi", new RepositoryProps
{
RepositoryName = "hl-m09-accounts-api",
ImageTagMutability = TagMutability.IMMUTABLE, // a pushed tag can never point at different bytes
ImageScanOnPush = true, // basic scanning, no extra charge
Encryption = RepositoryEncryption.AES_256, // use KMS with a customer managed key in production
RemovalPolicy = RemovalPolicy.DESTROY, // Learning shortcut: not for production
EmptyOnDelete = true, // lets 'cdk destroy' remove images
LifecycleRules = new[]
{
new LifecycleRule
{
RulePriority = 1,
Description = "Expire untagged images after 3 days",
TagStatus = TagStatus.UNTAGGED,
MaxImageAge = Duration.Days(3),
},
new LifecycleRule
{
RulePriority = 2,
Description = "Keep only the 15 most recent images",
TagStatus = TagStatus.ANY,
MaxImageCount = 15,
},
},
});
// Push role: narrow, one repository. In the lab this role is trusted via GitHub OIDC (see the OIDC lesson).
var push = new Role(this, "EcrPushRole", new RoleProps
{
RoleName = "hl-m09-ecr-push-example",
AssumedBy = new ServicePrincipal("codebuild.amazonaws.com"),
});
push.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Sid = "LogInToEcr",
Actions = new[] { "ecr:GetAuthorizationToken" },
Resources = new[] { "*" },
}));
// GrantPullPush would also grant pull actions and is broader than needed; write the push list explicitly.
push.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Sid = "PushToOneRepository",
Actions = new[]
{
"ecr:BatchCheckLayerAvailability", "ecr:InitiateLayerUpload", "ecr:UploadLayerPart",
"ecr:CompleteLayerUpload", "ecr:PutImage", "ecr:BatchGetImage",
"ecr:DescribeImages", "ecr:DescribeImageScanFindings",
},
Resources = new[] { Repo.RepositoryArn },
}));
// Inside a Stack, Amazon.CDK.Tags.Of(this) resolves to the stack's own property: qualify it.
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m09-delivery-pipeline");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
new CfnOutput(this, "RepositoryUri", new CfnOutputProps { Value = Repo.RepositoryUri });
}
}TagMutability.IMMUTABLE, ImageScanOnPush and LifecycleRules are the L2 construct properties for the three guardrails ⚠ VERIFY property names against the Amazon.CDK.AWS.ECR documentation for your Amazon.CDK.Lib version, particularly EmptyOnDelete.
A repository policy can add a second layer: for example, a statement that denies ecr:PutImage unless the caller is the push role, so even an administrator in another account cannot overwrite your images ⚠ design suggestion: test it in a non-production account first.
Complete code
The pipeline fragment that builds, scans and pushes, from the lab's release.yml:
- name: Build image locally first (scan BEFORE it reaches the registry)
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: .
load: true
push: false
tags: candidate:${{ github.sha }}
build-args: |
VERSION=${{ steps.meta.outputs.version }}
GIT_SHA=${{ github.sha }}
- name: Gate on CRITICAL vulnerabilities
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
with:
image-ref: candidate:${{ github.sha }}
severity: CRITICAL
ignore-unfixed: true
format: json
output: trivy-image.json
exit-code: "1"
- name: Push with immutable tags
id: push
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: .
push: true
provenance: mode=max
sbom: true
tags: |
${{ steps.ecr.outputs.registry }}/${{ vars.ECR_REPOSITORY }}:${{ steps.meta.outputs.git_tag }}
${{ steps.ecr.outputs.registry }}/${{ vars.ECR_REPOSITORY }}:${{ steps.meta.outputs.version }}
- name: Output the image reference by digest
id: digest
run: echo "image=${{ steps.ecr.outputs.registry }}/${{ vars.ECR_REPOSITORY }}@${{ steps.push.outputs.digest }}" >> "$GITHUB_OUTPUT"A C# helper to check that a task definition uses a digest, not a tag. Put it in a test project so a pull request that changes deploy/ecs/taskdef.json to a moving tag fails:
using System.Text.Json;
using System.Text.RegularExpressions;
using Xunit;
public partial class TaskDefinitionImageTests
{
// repository path, '@sha256:' and 64 hex characters, or a placeholder the pipeline replaces at deploy time.
[GeneratedRegex(@"^([a-z0-9._/-]+|[0-9]{12}\.dkr\.ecr\.[a-z0-9-]+\.amazonaws\.com/[a-z0-9._/-]+)@sha256:[0-9a-f]{64}$")]
private static partial Regex DigestRef();
[Theory]
[InlineData("111122223333.dkr.ecr.us-east-1.amazonaws.com/hl-m09-accounts-api@sha256:4b1f9c0e2d7a6b5c8e3f1a0d9c8b7a6f5e4d3c2b1a09f8e7d6c5b4a39281706f", true)]
[InlineData("111122223333.dkr.ecr.us-east-1.amazonaws.com/hl-m09-accounts-api:latest", false)]
[InlineData("111122223333.dkr.ecr.us-east-1.amazonaws.com/hl-m09-accounts-api:1.4.0", false)]
public void Deployed_images_are_referenced_by_digest(string image, bool expected)
=> Assert.Equal(expected, DigestRef().IsMatch(image));
[Fact]
public void Containers_use_a_read_only_root_filesystem_and_never_run_as_root()
{
using var doc = JsonDocument.Parse(File.ReadAllText("deploy/ecs/taskdef.json"));
foreach (var c in doc.RootElement.GetProperty("containerDefinitions").EnumerateArray())
{
Assert.True(c.GetProperty("readonlyRootFilesystem").GetBoolean());
// The chiseled image already runs as a non-root user. The task definition must not override it with root.
if (c.TryGetProperty("user", out var user))
{
Assert.NotEqual("root", user.GetString());
Assert.NotEqual("0", user.GetString());
}
}
}
}The lab's deploy/ecs/taskdef.json sets readonlyRootFilesystem and leaves user unset, so the image's non-root user applies.
Code explained line by line
load: true, push: false— builds into the local Docker image store so Trivy can scan it. Nothing has reached ECR, so a vulnerable image never becomes deployable.severity: CRITICAL,ignore-unfixed: true,exit-code: "1"— the gate. Fix what can be fixed; track what cannot.- Two tags in
tags:— the commit tag and the version tag. Both point at the same manifest; because ECR is immutable, neither can be moved later. provenance: mode=maxandsbom: true— BuildKit attaches attestations to the pushed image index. Pushing with attestations makes the image reference an index, and the digest you receive refers to that index, which is still immutable and usable by ECS ⚠ VERIFY that your Fargate platform version and ECS agent handle image indexes with attestation manifests without extra configuration.steps.push.outputs.digest— the digest of what was pushed. The job outputimagecomposesregistry/repository@digestand every later job uses that string. No later job reads a tag.- In the C# test, the regular expression requires
@sha256:followed by 64 hexadecimal characters; any tag form fails the test. GeneratedRegexis the source-generated regex in .NET; the class must bepartial. It compiles the pattern at build time and avoids startup cost.
Security considerations
⚠ Learning shortcut — not for production
The lab repository uses force_delete = true (Terraform) or EmptyOnDelete (CDK) so one command removes it, AES-256 encryption instead of a customer managed KMS key, and basic scanning only. In production: repositories are never force-deleted; images are encrypted with a customer managed key whose policy limits who can use it; enhanced scanning (continuous) is on for production repositories; and repository policies deny push from anything except the pipeline role.
- Supply chain. The image is only as trustworthy as its base image and packages. Pin the base by digest in production builds, mirror it with a pull-through cache or a copy in your registry so a public outage or a tampered tag cannot stop or poison builds, and have Dependabot raise reviewed updates.
- Least privilege at both ends. The push role cannot pull other repositories or delete images; the runtime role can only pull one repository.
- Non-root, read-only, no shell. The chiseled image has no shell; the task definition sets a read-only root filesystem; if the application must write, mount a temporary volume at that path. This limits what an attacker who gets code execution can do.
- No secrets in images. Anything in a layer can be extracted by anyone who can pull the image. Use BuildKit secret mounts for build-time secrets, and read runtime secrets from Secrets Manager (see Secrets in CI/CD).
- Scan results are time-sensitive. An image that was clean on Monday may have a critical CVE on Friday. Enhanced scanning and a scheduled rebuild catch this. Decide who is paged and how fast a critical must be patched; that is a policy question for your security team.
- Signing and verification. Provenance attestations and signatures only help if something verifies them before deploy. Decide where verification happens (admission control, a deploy-stage check) ⚠ VERIFY what ECS offers natively for verifying signatures, if anything.
- Data protection. Images contain code and configuration, not customer data. Keep it that way: never bake fixtures with real personal data into an image (UK GDPR). Test data in this module is fictional.
- Logging. ECR API calls appear in CloudTrail (
PutImage,BatchGetImage); an alarm onPutImagefrom a principal other than the push role detects misuse.
Banking use case
Harbourline's card-servicing team ships forty images a month. After an incident where a hotfix was "rebuilt" and behaved differently from the tested build, the change board introduced three rules, all implemented as pipeline code: one build per commit; production deploys reference a digest that was recorded in the test stage's evidence; and every image has an SBOM stored for seven years alongside the release record ⚠ retention period is a policy decision for your bank, not an AWS default. When a critical library vulnerability is announced, the security team queries the SBOM store for affected releases in minutes, and the platform team rebuilds those images through the same pipeline. All names and data are fictional.
Common mistakes
- Using
latest→ nobody can say what runs; rollbacks are guesses → tag with commit and version, deploy by digest. - Rebuilding for production → the image tested is not the image shipped → build once, promote the digest.
- Mutable tags → a later push silently changes what a tag means → set
IMMUTABLEon every repository. ARGfor tokens → credentials appear indocker history→ use secret mounts.- Wide push role (
ecr:*on*) → one compromised pipeline overwrites any image → list the push actions, one repository ARN. - Root user and full SDK image at runtime → large attack surface and an unnecessary shell → multi-stage, chiseled, non-root.
- Copying the whole repository before restore → every code change reinstalls all packages → copy project files first, restore, then copy source.
- Lifecycle policy that deletes the image in production → a rollback target disappears → keep at least the last several tagged releases; preview the policy before applying.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
no basic auth credentials on push | Not logged in, or token expired (12 hours) | Run aws ecr get-login-password ... again |
denied: User ... is not authorized to perform: ecr:InitiateLayerUpload | Push role lacks push actions or the resource ARN is wrong (Region, account, name) | Compare with permissions-ecr-push.json |
tag invalid: The image tag '1.4.0' already exists | Immutable tag | Bump the version; never re-push |
name unknown: The repository with name ... does not exist | Wrong Region or repository name, or repository not created | Check --region, create via IaC |
ECS task fails with CannotPullContainerError | Task execution role cannot pull, or no route to ECR (private subnet without endpoints or NAT) | Fix the execution role; add VPC endpoints for ECR and S3, or a NAT gateway |
| Container exits at once with a permission error on a path | Read-only root filesystem, process writes to disk | Mount a temporary volume at that path or remove the write |
exec format error at start | Image architecture does not match the task's CPU architecture | Build for the right --platform; set runtimePlatform in the task definition |
| Trivy fails the build on a new CVE with no code change | Base image or package vulnerability published | Rebuild on the patched base; if none exists, record a time-limited exception |
| Build is slow on every run | Restore layer invalidated each time | Copy only project files before dotnet restore; use the cache backend |
Cost and cleanup
Cost: credits (well under 1 US dollar if cleaned up). ECR storage is billed per GB-month, with a small free tier for new accounts ⚠ VERIFY current storage pricing and free tier; the lab image is around 100 MB, so storage cost is negligible, but data transfer out of the Region and pulls by NAT gateways in your VPC can cost more than the storage. Basic scanning on push has no extra charge; enhanced scanning is billed by Amazon Inspector per image scanned ⚠ VERIFY. The lifecycle policy keeps storage bounded.
Free alternative: build and run the image locally and skip the push; the scan, tag policy and Dockerfile exercises all work without an AWS account.
Teardown:
P=learn-sandbox
# Delete the repository and every image in it. Irreversible; the lab data is fictional.
aws ecr delete-repository --repository-name hl-m09-accounts-api --force --profile $P --region us-east-1
# Or, if created with Terraform:
# cd infra/terraform && terraform destroy
# Remove local images.
docker image rm hl-m09-accounts-api:dev 2>/dev/null; docker image prune -f
# Confirm nothing remains.
aws ecr describe-repositories --query "repositories[?starts_with(repositoryName, 'hl-m09-')].repositoryName" --profile $P --region us-east-1
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m09-delivery-pipeline --profile $P --region us-east-1Exercises
- 🟢 Guided: Build the lab image locally with
VERSION=1.0.0and your current commit, run it, confirm/versionshows the commit, then run it withHealth__ForceUnready=trueand confirm/health/readyreturns 503. Runimage-tagswith a short SHA and note the error. - 🟡 Challenge: Add a
.dockerignore-aware check: write a small test or script that fails if the Dockerfile containsARGnames that look like secrets (TOKEN,PASSWORD,KEY,SECRET) or if the final stage lacks a non-rootUSER. Then extend the lifecycle policy so that images tagged with a semantic version are kept for 365 days whilegit-tagged images expire after 30 days. - 🔴 Live problem: After a Friday release, production runs a container that behaves differently from test, and nobody is sure which image it is. The task definition says
hl-m09-accounts-api:1.4.0, and a colleague admits re-pushing that tag from a laptop "to fix a typo" last week. Explain what went wrong at each layer, how to find out what is really running, and what you change so it cannot recur.
Assignment
Add docs/image-policy.md to your portfolio repository: your tagging rules, the digest-only deployment rule, the ECR repository settings (immutability, scanning, lifecycle, encryption), the exact IAM actions for push and for pull with a one-line reason for each, and a table of the scans that run at each stage with the gate you chose and why.
Solutions
docker build --build-arg VERSION=1.0.0 --build-arg GIT_SHA=$(git rev-parse HEAD) -t hl-m09-accounts-api:dev ., then docker run --rm -p 8080:8080 hl-m09-accounts-api:dev. curl localhost:8080/version returns the version and commit, because the Dockerfile stamps Version and SourceRevisionId into the assembly. With -e Health__ForceUnready=true, /health/ready returns 503 while /health/live still returns 200; the load balancer would stop sending traffic, which is the signal blue/green and alarms rely on. A short SHA is rejected by ImageTagPolicy because an abbreviated hash can be ambiguous and is not what CI provides: use the full 40 characters and let the tool shorten it.
A simple check: read the Dockerfile, fail if any ARG or ENV name matches a case-insensitive pattern for TOKEN|PASSWORD|SECRET|KEY, and fail if the last FROM stage has no USER instruction or sets USER root. As a test it can be a few lines of C# with File.ReadAllLines and a regex. For the lifecycle policy, ECR rules evaluate by priority and tag prefix: rule 1 for tag prefix git- (tagStatus: tagged, tagPrefixList: ["git-"], sinceImagePushed 30 days, expire); rule 2 for tag prefix list covering version tags (semantic versions start with digits, so use the digits 0 to 9 as prefixes ⚠ VERIFY) with 365 days; rule 3 for untagged after 3 days. Preview with start-lifecycle-policy-preview before applying. Note that an image carrying both a git- tag and a version tag matches both rules; check how ECR resolves overlap before relying on a shorter expiry for one tag ⚠ VERIFY, otherwise the git- rule could expire a released image.
What went wrong: (1) a mutable tag, so 1.4.0 stopped meaning one set of bytes; (2) a human could push from a laptop, so the registry accepted images from outside the pipeline; (3) the task definition referenced a tag, so ECS may have started tasks from different content depending on when each task started. ECS resolves tags to digests when a deployment starts and then keeps that digest for the service revision, so which digest is running depends on when the deployment happened ⚠ VERIFY. To find what runs: aws ecs describe-tasks shows containers[].imageDigest for each running task; compare with the digests recorded in the test stage's evidence and in ECR (describe-images, imagePushedAt, and CloudTrail PutImage events showing who pushed). Fixes: set IMMUTABLE on the repository; remove push permission for humans (only the OIDC push role may push, enforce with a repository policy); deploy by digest only (the unit test above); record the digest in the release evidence. Then explain the incident to the change board as a control failure with a corrective control, not as a person's mistake.
Knowledge check
Answer: A digest identifies exact content and can never change. A tag can be re-pointed unless the repository is immutable, and even then humans mistake it for a version. The digest recorded in test evidence proves the deployed image is the tested one.
Answer: ecr:GetAuthorizationToken; it returns a registry login token and has no resource-level scope. It matters little because the token only authorises what the caller's other permissions allow, which you limit by repository ARN.
Answer: It stops a tag being re-pointed to different content. It does not stop someone pushing a new tag, deleting an image, or deploying a vulnerable image; scanning, IAM and approvals cover those.
Answer: So a vulnerable image never reaches the registry where it could be deployed by mistake or pulled by another team. Post-push scanning (basic or enhanced) then catches CVEs published later.
Answer: Fewer packages mean fewer vulnerabilities; no shell or package manager limits what an attacker can do after exploiting the app; non-root limits access to the container's resources; the image is also smaller and starts faster.
Answer: Docker caches layers by their inputs. If only project and props files are copied before restore, the restore layer is reused until dependencies change, so normal source edits do not trigger a full package download.
Interview questions
- How do you make sure what you tested is what you deploy? Build once, push with immutable tags, record the digest, and reference the digest in every environment's task definition. Production never rebuilds.
- What would you put in a push role for ECR?
GetAuthorizationTokenon all resources and the layer-upload,PutImageand read actions on one repository ARN; no delete, no policy or lifecycle changes, no repository creation. - How would you handle a new critical CVE in a base image? Look up affected releases in the SBOM store, rebuild each image on the patched base through the normal pipeline, scan, test, and promote by the normal approval; do not patch running containers.
- Basic versus enhanced ECR scanning? Basic scans on push (and on demand) with a standard CVE database. Enhanced uses Amazon Inspector for continuous scanning including language packages, billed separately.
- Why not use
latest? It is a moving pointer, so rollbacks, audits and incident reports cannot say which build ran. It also hides redeploys: the task definition does not change when the image does.
Certification practice
A team's production ECS task definition references app:latest. After an incident they cannot tell which build was running. Which change gives full traceability and guarantees that the tested image is deployed?
A. Enable ECR scan on push.
B. Make the repository tags immutable and reference the image digest in the task definition, recording it in release evidence.
C. Add a lifecycle policy that keeps only one image.
D. Deploy latest but add a GIT_SHA environment variable.
Answer: B. The digest names exact content and immutability prevents tag drift. A finds vulnerabilities but does not identify builds. C removes rollback targets. D records a label that may not match the bytes, and the task definition still does not change when the image does.
A CI role must push images to a single ECR repository and nothing else in ECR. Which policy design is correct?
A. ecr:* on all resources.
B. ecr:GetAuthorizationToken on all resources plus the layer-upload and ecr:PutImage actions on the repository's ARN.
C. ecr:GetAuthorizationToken restricted to the repository ARN.
D. Only ecr:PutImage on the repository ARN.
Answer: B. The token action does not support resource-level restriction, so it needs *; the push actions are limited to the repository. A is far too broad. C will not work because the action ignores resource ARNs. D omits login and layer upload, so pushes fail.
A pipeline job fails to push orders-api:2.3.0 with "tag invalid: The image tag already exists in the repository and cannot be overwritten". The code did not change. What happened and what is the correct fix?
A. The registry is full; delete old images.
B. The repository has tag immutability on and the version was already pushed; bump the version for a new build.
C. The IAM role lacks ecr:PutImage; add it.
D. The scan on push failed; disable it.
Answer: B. The message is the immutability control working. C would give an authorisation error. A and D are unrelated to this error and weaken controls.
Completion checklist
- I can explain the difference between a tag and a digest in two sentences
- I read the lab Dockerfile and can justify every instruction
- I created or reviewed an ECR repository with immutability, scan on push and a lifecycle policy
- I can list the ECR actions a push role needs and which one needs
* - I ran the cleanup steps and confirmed no repository or image remains
Key takeaways
- Build once, tag with commit and version, and deploy by digest; never use
latest. - Enforce the tagging policy in the registry with immutable tags, and keep the push role to one repository.
- Use a multi-stage, chiseled, non-root image; scan the candidate before pushing and keep scanning afterwards.
- Attach provenance and an SBOM so you can answer "what is in production and where did it come from?"
- Lifecycle policies control cost, but preview them so they never delete a rollback target.
Further reading
- Amazon ECR image tag mutability
- Scanning images for software vulnerabilities in Amazon ECR
- Enhanced scanning for Amazon ECR
- ECR lifecycle policies
- Private registry authentication in Amazon ECR
- Pull through cache rules
- Private image replication
- Build secrets (Docker Docs)
- GitHub Actions cache backend for BuildKit (Docker Docs)
- Next: Automated testing strategy
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
AWS CodeBuild, CodePipeline and CodeDeploy
Build a .NET 10 image in CodeBuild with a buildspec, orchestrate it with a CodePipeline V2 pipeline that has a manual approval, understand where CodeDeploy fits, and decide when AWS-native tooling beats GitHub Actions.
Automated testing strategy for pipelines: unit, integration, contract, end-to-end and load
Decide which tests run at which pipeline stage, write fast unit tests, in-process integration tests, Testcontainers tests against PostgreSQL and LocalStack, a consumer contract test, a post-deploy smoke test and a k6 load test, and keep the whole suite trustworthy and quick.