Module project: complete CI/CD for an ASP.NET Core API
Build, scan and sign off a release of the Harbourline Accounts API end to end: OIDC everywhere, one image built once and deployed by digest to test and then, after a human approval and a release-policy check, to production with ECS blue/green and alarm rollback, ending in an append-only evidence pack.
Assumes: .NET 10 LTS ยท GitHub Actions (configure-aws-credentials v6.3.0, amazon-ecs-deploy-task-definition v2.6.4, trivy-action v0.36.0) ยท Amazon ECS native blue/green ยท Terraform 1.10 or later (AWS provider 6.x) ยท AWS CLI v2 ยท Region us-east-1 ยท labs/m09-delivery-pipeline. Verified against official sources on 2026-11-18.
What you are building
A delivery pipeline that a bank's internal audit would accept as a control, not just a convenience. For the Module 3 Accounts API it will:
- build once, test, scan and push an immutable image to Amazon ECR;
- deploy that exact image, referenced by digest, to a
testenvironment; - wait for a human approval on a protected
prodenvironment, made by someone other than the author; - check the release against a written release policy (ticket, digest, scans, segregation of duties);
- deploy to production with ECS blue/green, where a CloudWatch alarm or a failed health check rolls traffic back automatically;
- store a tamper-resistant evidence record for the release.
No AWS access key exists anywhere in the design. All data and names are fictional.
Learning objectives
By the end of the project you can:
- Operate a pipeline in which each job receives short-lived credentials for one role, scoped by repository and environment.
- Promote one image by digest through environments without rebuilding it.
- Configure GitHub environments with required reviewers, prevent self-review and a branch restriction.
- Rehearse automatic rollback by breaking the readiness check on purpose, and explain exactly what ECS did.
- Produce and verify a release-evidence record that links ticket, commit, reviewers, scans, approvals and image digest.
- Tear everything down and prove nothing is billing.
Prerequisites
- GitHub Actions with OIDC and Lab 9.2, which creates the provider and roles used here.
- Lab 9.1, for the
ciworkflow. - ECS deployment strategies, Rollback strategies and Release management and audit evidence.
- Secrets in CI/CD, for why this pipeline holds no runtime secrets.
- The ECS on Fargate stack from Module 3 (
labs/m03-accounts-api/infra/cdk) to host the service. - Tools: Docker, AWS CLI v2 with the
learn-sandboxprofile, Terraform 1.10 or later, a GitHub repository you own.
Cost label
| Part | Label | Notes |
|---|---|---|
Lab 9.1 and 9.3 workflows, trust-*, gate and evidence tools | ๐ข free | Public repositories: free runner minutes. Private: monthly allowance โ VERIFY |
| OIDC provider, IAM roles, small ECR repository, evidence bucket | ๐ข free to a few cents | Destroyed at the end |
| ECS Fargate service plus Application Load Balancer for the blue/green exercise | ๐ก credits | Roughly an hour of two small Fargate tasks plus an ALB: well under a dollar, but price it yourself in the AWS Pricing Calculator โ VERIFY; destroy the same day |
| Lab 9.5 Lambda canary | ๐ข free | Within the always-free allowance |
| Lab 9.6 CodeBuild and CodePipeline | ๐ก credits | Optional; V2 pipelines bill per action minute, with a monthly free quota โ VERIFY |
Free alternative: do everything except the Fargate deployment. The release-policy tool, health gate, trust evaluation and evidence record all run offline, and you can read the blue/green configuration as a design exercise.
Architecture
Trust boundaries
Each arrow exists because a trust policy names that exact subject. There is no arrow from ecr-push to a deploy role, and none from a fork anywhere.
Microsoft / Azure comparison
| You know (Azure DevOps) | This project | What's different |
|---|---|---|
| Multi-stage YAML pipeline with stages | Jobs in release.yml with needs | Same idea; jobs run on fresh runners |
| Environments with approvals and checks | GitHub environments with required reviewers | "Prevent self-review" is a built-in setting |
| Service connection to Azure | OIDC role per environment | No secret exists |
| Deployment slots and swap | ECS blue/green with test listener and bake time | Traffic is shifted by the ALB; rollback is also automatic on alarm |
| Pipeline artefact "drop" | Image digest plus build-evidence artefact | Deploy by digest, not by build number |
| Release gates (query monitor, work item) | Release-policy check and CloudWatch alarms | Written as code and tested |
Part 1: the AWS side (Lab 9.2, Part B)
Apply the Terraform
cd labs/m09-delivery-pipeline/infra/terraform # use your copy
cp backend.hcl.example backend.hcl
cp terraform.tfvars.example terraform.tfvars
terraform init -backend-config=backend.hcl
terraform plan -out tf.plan
terraform apply tf.plan
terraform outputYou get the OIDC provider, hl-m09-gh-ecr-push, hl-m09-gh-deploy-test, hl-m09-gh-deploy-prod, the ECR repository hl-m09-accounts-api (immutable tags, scan on push, lifecycle policy) and the evidence bucket hl-m09-release-evidence-<account id> (versioning, KMS, public access blocked, TLS-only policy, Object Lock in GOVERNANCE mode for one day).
Host the service
The deploy roles may update only service/hl-m09-cluster/hl-m09-accounts-api and pass only the roles hl-m09-accounts-api-task and hl-m09-accounts-api-execution. Deploy Module 3's Fargate stack with those names, with an ALB that has a production listener and a test listener and two target groups, hl-m09-blue and hl-m09-green. Then switch the service to the native blue/green strategy using the lab's configuration:
aws ecs update-service --cli-input-json file://deploy/ecs/service-deployment.json --profile learn-sandboxEdit the placeholder ARNs in that JSON first (target groups, listener rules and the infrastructure role). The file selects BLUE_GREEN, a 10-minute bake time, two alarms with rollback enabled (hl-m09-accounts-api-5xx and hl-m09-accounts-api-p99-latency), and a PAUSE lifecycle hook at POST_TEST_TRAFFIC_SHIFT. Create those two CloudWatch alarms before you update the service. โ VERIFY the field names and the ALB requirements against the ECS API reference and ECS deployment strategies before applying: this feature is recent and the console may show a different layout.
โ Learning shortcut โ not for production
Test and prod share one account, one cluster name and the same permissions. Production uses its own AWS account, cluster, alarms and role, and an SCP that blocks workload changes except by the pipeline. The same service-deployment.json is applied in both; only the ARNs differ.
Part 2: the GitHub side
Create the repository
Copy the contents of the lab folder to the root of a new repository (workflows must be in .github/workflows at the root). Enable the repository rules from Git and GitHub for regulated teams: a ruleset on main requiring a pull request, one approving review, code-owner review, required status checks (build-and-test, container-build, iam-policy-lint) and no force pushes. The lab's .github/CODEOWNERS names who must review pipeline, IAM and release-rule changes; replace the placeholder owners with real accounts or teams.
Create the environments
In Settings โ Environments create test and prod.
| Setting | test | prod |
|---|---|---|
| Required reviewers | none | at least one person who is not a typical author |
| Prevent self-review | n/a | on |
| Deployment branches | main | main only |
| Variables | AWS_DEPLOY_ROLE_ARN, ECS_CLUSTER, ECS_SERVICE, SMOKE_URL | the same four plus EVIDENCE_BUCKET, CHANGE_TICKET_PATTERN |
Environment protection rules on private repositories may require a paid plan โ VERIFY your plan. Repository-level variables: AWS_REGION, ECR_REPOSITORY, ECR_PUSH_ROLE_ARN, ECR_REGISTRY. Use the Terraform outputs for the role ARNs. Role ARNs are variables, not secrets: they identify a role, and seeing them in logs helps auditors.
Read release.yml as a design
The file's top comment is the architecture. Four decisions deserve attention.
| Decision | Where | Why |
|---|---|---|
permissions: contents: read by default, id-token: write only on jobs that assume a role | top and per job | Least privilege for GITHUB_TOKEN and OIDC |
concurrency: group: release, cancel-in-progress: false | top | Two releases never overlap, and a running deployment is never cancelled half-way |
Scan before push: build with load: true, push: false, run Trivy with severity: CRITICAL, ignore-unfixed: true, exit-code: "1" | build-push | A critical, fixable vulnerability never reaches the registry |
deploy-prod has if: github.event_name == 'workflow_dispatch' | deploy-prod | Pushes to main flow only as far as test. A production release is a deliberate, ticketed run with two inputs: change_ticket and version |
Follow the image from commit to digest
In build-push the job: computes the tags with the lab's own image-tags tool (git-<12 hex chars> and the semantic version, never latest); assumes the ECR push role; logs in to ECR; builds and scans locally; pushes with provenance: mode=max and sbom: true; and then publishes the digest-pinned reference as the job output image. Every later job deploys repository@sha256:<digest>, so test and prod run byte-identical images. It also creates a GitHub build-provenance attestation (actions/attest-build-provenance) and an SPDX SBOM (anchore/sbom-action), and uploads the Trivy report and SBOM as the build-evidence artefact for 90 days.
Why digest and not tag: a tag is a label somebody can move; ECR immutability prevents that for you, but a digest is also verifiable by anyone downstream and is what ECS records as the running image.
Deploy to test
deploy-test declares environment: test, so the OIDC subject becomes repo:<owner>/<repo>:environment:test, which only hl-m09-gh-deploy-test trusts. It renders deploy/ecs/taskdef.json with the digest image and build metadata, registers it, updates the service and waits (up to 20 minutes) for stability. Then it checks that the running version is the commit we built:
curl -fsS --retry 10 --retry-delay 6 --retry-all-errors "$SMOKE_URL/version"
# {"version":"1.4.0","commit":"3f9c2ab1...","environment":"test","releaseId":"accounts-api-1.4.0"}The workflow greps for "commit":"<sha>". A deployment that succeeded in ECS but is running old code fails here.
Approve and deploy to prod
Trigger a release:
GitHub โ Actions โ release โ Run workflow, inputs: change_ticket = CHG-104233, version = 1.4.0.
The deploy-prod job stops at "Waiting for review". The reviewer sees the environment, the run and the commit. After approval the job:
- downloads the Trivy report and SBOM;
- assembles the evidence with the GitHub API: the pull request that produced the commit, who authored and merged it, who approved the PR, the run's approvals, and the Trivy critical and high counts;
- runs
dotnet run --project src/Harbourline.Delivery -- evidence evidence-out/release.json, the release policy, which fails the job on any violation; - assumes the prod role through OIDC, renders the task definition with the same digest, and updates the service. ECS performs the blue/green deployment, shifts traffic, and bakes;
- smoke-tests
/versionagainst production; - in an
if: always()step writes the evidence record to S3 with--if-none-match '*', so an existing record is never overwritten.
Part 3: the release policy, offline
The evidence command encodes the rules from the release-evidence lesson. Run it against the sample record:
dotnet run --project src/Harbourline.Delivery -- evidence samples/release-evidence.jsonRelease accounts-api-2026.11.18-1 sha256=2083e43c964ef975014d16f0b08f2cfc5bfc89e999ff965d70972f4c65f5df8d
Release policy satisfied.Now break it. Change "approver": "grace.adeyemi" to "priya.shah" (the commit author) and run again:
Release accounts-api-2026.11.18-1 sha256=...
VIOLATION: The commit author approved their own production release.
Release blocked.The sha256 changes with the content: the record's hash is stored with the object, so a later edit is detectable. The policy rejects: no change ticket, a short SHA, an image referenced by tag, failed tests, zero tests, critical findings without a listed risk acceptance, no prod approval, a PR with no reviewer other than the author, and an author approving their own release.
Where the policy comes from
These rules are an example. Whether one person may both merge and approve a production release, how many reviewers are required, and what counts as an accepted risk are decisions for your bank's change-management policy, risk and audit teams. Never claim a pipeline makes you compliant; it produces evidence for people who decide that.
One known simplification in release.yml: the tests block of the evidence is a placeholder (passed: 1). The ๐ก exercise below replaces it with the real counts from the test-results artefact.
Part 4: blue/green, and breaking it on purpose
Rehearse the failure
The API exposes /health/live and /health/ready. Readiness reads the setting Health__ForceUnready; when it is true, readiness returns 503 while liveness stays 200 (liveness must not fail or ECS would restart-loop the task).
-
Locally, prove the behaviour:
docker run --rm -p 8080:8080 -e Health__ForceUnready=true hl-m09-accounts-api:dev curl -i localhost:8080/health/ready # 503 curl -i localhost:8080/health/live # 200 -
In your repository change the
Health__ForceUnreadyvalue indeploy/ecs/taskdef.jsonto"true"on a branch for the test environment only, merge by pull request, and run the release. In ECS you will see the green tasks start, the target group mark them unhealthy, and the deployment fail without production traffic ever reaching the bad version. For prod, the same change would be caught by the alarms and the bake time. In both cases blue keeps serving. -
Revert the setting by a new pull request and a new version. Because tags are immutable, you cannot re-push
1.4.0; releasing1.4.1is the honest, auditable path. -
Read the history: Console โ ECS โ Clusters โ your cluster โ Services โ Deployments, or:
aws ecs list-service-deployments --cluster hl-m09-cluster --service hl-m09-accounts-api --profile learn-sandboxโ VERIFY the CLI command names in your CLI version; the service-deployment APIs are recent.
The decision logic of a rollback gate is also available offline as the gate tool, which turns health samples into Proceed, Hold or Rollback:
dotnet run --project src/Harbourline.Delivery -- gate samples/canary-failing.jsonDecision: Rollback
- Error rate 10.30 % exceeds 1.00 %.
- p99 latency 1020 ms exceeds 800 ms.and samples/canary-healthy.json gives Decision: Proceed with Error rate 0.00 %, p99 295 ms โ within policy. Thresholds (1% errors, 800 ms p99, at least 50 requests for a verdict, two consecutive bad samples) are written down deliberately: they are the objective go or no-go criteria a change board wants to read. See Rollback strategies for why database changes can make "roll back" unsafe.
Lab 9.3: scanning and SBOM (security.yml)
The security workflow runs on pull requests, on main and weekly (Mondays 06:17 UTC), because new vulnerabilities appear for code that has not changed.
| Job | Tool | Gate |
|---|---|---|
secrets | Gitleaks over the full history (fetch-depth: 0) | Any secret fails |
dependencies | NuGet Audit (restore) plus dotnet package list --vulnerable --include-transitive | High and critical fail the restore |
image | docker build, SPDX SBOM (Anchore), Trivy table report, Trivy JSON gate | Fixable CRITICAL fails |
Trivy is pinned to v0.36.0 by SHA, with a comment in the workflow: the lab never uses a trivy-action tag older than v0.35.0 because of a compromise of that action in March 2026 โ VERIFY the incident details in the project's advisory before citing them. This is the practical lesson in why you pin actions. Prove the gate: add an obviously fake AWS-style key to a file on a branch and watch the secrets job fail, then remove it and treat the key as burned (history keeps it). Use only the documented example key from AWS docs, never a real one. For tool depth see Pipeline security scanning in Module 5 and Pipeline security.
Lab 9.5: Lambda canary (design and CLI)
deploy/lambda/appspec.yml is the raw form of what AWS SAM generates for you. It names the function, the alias live, the current and target versions, and two hooks (BeforeAllowTraffic, AfterAllowTraffic). The deployment group uses the configuration CodeDeployDefault.LambdaCanary10Percent5Minutes (10% of traffic for five minutes, then the rest) and an alarm on hl-m09-statements-errors.
# After publishing version 4 of the function and creating the deployment group:
aws deploy create-deployment \
--application-name hl-m09-statements \
--deployment-group-name hl-m09-statements-canary \
--revision '{"revisionType":"AppSpecContent","appSpecContent":{"content":"<appspec as a JSON string>"}}' \
--profile learn-sandboxโ VERIFY the exact --revision structure for Lambda in the CodeDeploy CLI reference; most teams let SAM or CDK generate and run this. The full walkthrough, hooks and rollback behaviour are in Lambda deployments. To rehearse: make version 4 return errors and invoke the alias in a loop. The alarm trips during the five canary minutes and CodeDeploy moves the alias back to version 3. Cost is free within the always-free tier; delete the function, alias, alarm and deployment group afterwards.
Lab 9.6: the same build in CodeBuild (optional)
buildspec.yml repeats the pipeline's build in AWS CodeBuild so you can compare the two models. It runs the tests inside the official mcr.microsoft.com/dotnet/sdk:10.0 container (so results do not depend on which .NET the CodeBuild image ships), computes tags with the same image-tags tool, builds and pushes, resolves the digest, exports IMAGE_URI, writes imagedefinitions.json for the CodePipeline ECS deploy action, and publishes JUnit reports.
Requirements: the standard Amazon Linux image (the file says aws/codebuild/amazonlinux-x86_64-standard:5.0 or newer โ VERIFY the current image name), privileged mode enabled for Docker builds, a service role limited to ECR push for one repository plus its own logs and report groups, and plain-text variables ECR_REGISTRY, ECR_REPOSITORY and APP_VERSION. In CodePipeline V2: Source (GitHub through CodeConnections โ VERIFY current naming) โ Build (CodeBuild) โ Manual approval โ Deploy (Amazon ECS). Differences to note: the approval lives in CodePipeline and is recorded there; identity is the service role, not OIDC; and cost is per action minute. See CodeBuild, CodePipeline and CodeDeploy.
Security considerations
โ Learning shortcut โ not for production
GOVERNANCE-mode Object Lock for one day lets an administrator with the bypass permission delete evidence, and force_destroy removes the bucket in the lab. A bank sets COMPLIANCE mode with a retention period from its records policy, restricts who can change the bucket policy, and replicates evidence to a separate log-archive account.
- No long-lived credentials. No IAM user, no key, no secret holding AWS access. If you find one, delete it and rotate.
- Least privilege per job. The ECR role pushes to one repository; deploy roles update one service and pass two roles to ECS; only prod writes under
releases/*in the evidence bucket. - Segregation of duties is enforced twice. By GitHub (prevent self-review, required reviewers) and by the release policy (author is not the approver).
- Untrusted code never gets a role. Pull-request workflows have no
id-tokenpermission. - Dependencies and actions are pinned by SHA and updated by reviewed Dependabot PRs.
- Evidence is append-only.
If-None-Match: *refuses overwrite, versioning keeps history, Object Lock prevents deletion for the retention period, and each record carries a SHA-256. - Runtime secrets bypass the pipeline. The service reads its own secrets through its task roles; see Secrets in CI/CD.
Common mistakes
- Rebuilding the image for prod "to be safe". That destroys the claim "what we tested is what we shipped".
- Approving a production deployment on mobile without reading the commit and ticket.
- Alarms with
Missingdata treated as healthy, so a dead service looks fine โ choose the treat-missing-data setting deliberately. - Setting a bake time shorter than the metric period of the alarms, so an alarm cannot fire before the old tasks are gone.
- Re-pushing a tag after a failed release. Immutable tags forbid it: bump the version.
- Storing evidence only as workflow logs. Logs expire and are editable by admins; export the record.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
build-push fails at "Assume the ECR push role" | Job is not on main, so sub does not match the ECR trust policy | Run from main, or check the trust policy subject |
| Trivy step fails the build | A fixable CRITICAL vulnerability | Update the base image or package; do not add an ignore without a recorded risk acceptance |
tag invalid: The image tag already exists | Immutable tags and a repeated version | Use a new version |
deploy-test hangs then times out | New tasks fail the target group health check | Check /health/ready, security groups, task logs in /ecs/hl-m09-accounts-api |
deploy-prod never starts | Waiting for approval, or workflow_dispatch was not used | Approve in the run page, or start the workflow manually |
| Release policy fails with "No tests ran" | The placeholder or empty test summary | Fill in real counts; a green build with zero tests is not evidence |
Evidence step fails with PreconditionFailed | A record for that release and status already exists | Intended: bump the version, or investigate why a duplicate was attempted |
| ECS rolled back and the workflow is red | Alarm or health failure during the deployment | Read the service deployment events; fix forward with a new version |
iam:PassRole denied | The task role names in the policy differ from your stack | Align the names in oidc.tf and the Module 3 stack |
Cost and cleanup
Destroy the ECS and ALB resources the same day, then the lab Terraform.
# 1. Module 3 stack (ECS service, ALB, alarms) โ from labs/m03-accounts-api/infra/cdk
cdk destroy --profile learn-sandbox
# 2. This lab's IAM, ECR and evidence bucket
cd labs/m09-delivery-pipeline/infra/terraform
terraform destroy
# Evidence objects are locked for 1 day (GOVERNANCE). If destroy reports locked versions, wait a day
# or remove them with --bypass-governance-retention (needs s3:BypassGovernanceRetention).
# 3. Confirm nothing from the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m09-delivery-pipeline --profile learn-sandbox --region us-east-1
aws iam list-roles --query "Roles[?starts_with(RoleName, 'hl-m09-')].RoleName" --profile learn-sandboxAlso delete the CloudWatch log group /ecs/hl-m09-accounts-api, any Lambda function, CodeDeploy application and CodeBuild or CodePipeline resources from Labs 9.5 and 9.6, and the environment variables in GitHub, so a future run fails fast instead of half-way.
Exercises
- ๐ข Guided: run the release workflow to test only (a push to
main), then a manual release with a ticket. Download the evidence JSON from S3 and run theevidencetool against it. - ๐ก Challenge: replace the placeholder test counts. In
deploy-prod, download thetest-resultsartefact of thecirun for the same commit (viagh run downloador the artefacts API), count passed, failed and skipped from the JUnit XML withjqor a small .NET tool, and put them in the evidence. Add a failing-test fixture to the unit tests of the policy. - ๐ด Live problem: auditors ask, "Show me every production change in November, who approved it, and prove the image that ran is the one that was tested." You have 30 minutes. Describe which records you pull, from where, and what you cannot prove today. Then list the three improvements you would make first.
Assignment
Publish your repository (public, or private with screenshots) containing: the workflows, the policies and the Terraform; a RELEASE-EVIDENCE.md that explains the evidence record field by field; one real evidence JSON from your run; and a short design note answering "what would change in a real bank?" (separate accounts, SCPs, COMPLIANCE-mode retention, change-management integration, who approves).
Solutions
A push to main runs build-push and deploy-test only; deploy-prod is skipped because the event is not workflow_dispatch. The manual run adds the approval wait, the policy step, the blue/green deployment and the evidence upload. The evidence key looks like releases/hl-m09-accounts-api/2026/accounts-api-1.4.0-success.json; the job status is part of the name, so a failed release is recorded too. Running the evidence command on the downloaded file prints the same SHA-256 you can compute with sha256sum only if you canonicalise the JSON the same way, so compare the stored sha256 metadata on the object instead.
Add a step before the evidence assembly: gh run list --workflow ci.yml --commit "$GITHUB_SHA" --json databaseId --jq '.[0].databaseId', then gh run download <id> -n test-results -D tests-in. Parse each JUnit file: with jq this is awkward, so use xmllint --xpath or a short dotnet script to total the tests, failures and skipped attributes. Pass the three numbers as --argjson values into the existing jq -n that builds release.json. Why this matters: the policy is only as good as its inputs. A hard-coded passed: 1 proves nothing. Add a unit test where Tests.Passed = 0 expects "No tests ran".
Pull: GitHub deployment history for the prod environment and the evidence objects under releases/<service>/2026/ (the November ones), CloudTrail events for UpdateService and RegisterTaskDefinition by the prod role, ECS service deployment history, and the ticket system. Chain each release: ticket, PR, commit, image digest, approval, deployment. To prove the image ran, show the task definition's digest equals the evidence record and the ECR image push provenance. Gaps typically found: placeholder test counts, ticket numbers not validated against the ticket system, evidence in the same account as the pipeline, no alarm history. First improvements: validate the ticket through an API, store evidence in a separate log-archive account in COMPLIANCE mode, and export approval and deployment events automatically.
Knowledge check
Answer: So the artefact that passed tests, scans and the test environment is byte-identical to what goes live. Rebuilding can pick up new dependencies or base-image layers and invalidates the evidence trail.
Answer: Two release runs never deploy at once, and a deployment already in progress is never cancelled mid-way, which could leave the service in an unknown state. Queued runs wait their turn.
Answer: GitHub's "prevent self-review" setting on the prod environment, and the release policy check, which fails when an approver equals the commit author. Two independent controls mean one misconfiguration is not enough.
Answer: After production traffic has shifted to green and the bake time has passed without an alarm firing or targets failing, the old (blue) tasks are scaled down. Until then, rollback is a traffic shift, not a redeploy.
Interview questions
- Walk me through a production release in your pipeline. Merge to protected main after review; build once; scan; push by digest; deploy to test; approval on the prod environment by someone other than the author; release-policy check; blue/green deploy with alarm rollback; evidence stored immutably.
- How do you prove the image in production is the one you tested? The digest in the evidence record, the task definition and ECR match; plus the provenance attestation and SBOM tied to that digest.
- A deploy failed and rolled back. What do you do next? Confirm the service is healthy on the previous version, capture the failure evidence, fix forward with a new version through the normal pipeline, and record the incident. Never re-push an existing tag.
Certification practice
A team wants production deployments to roll back automatically when the error rate rises after traffic shifts to a new ECS service revision. Which configuration achieves this? A. Increase the task count. B. Use blue/green deployment with CloudWatch alarms configured on the service deployment and rollback enabled. C. Re-run the pipeline manually. D. Use a larger Fargate task size. Answer: B. Alarms watched during the deployment and bake time trigger an automatic traffic shift back to the previous revision. A and D change capacity, not safety; C is manual.
Which image reference gives the strongest guarantee that test and production run identical code? A. repo:latest B. repo:1.4.0 in a repository with mutable tags C. repo@sha256:<digest> D. repo:prod
Answer: C. A digest identifies exact content. Mutable tags such as A, B and D can be repointed.
Evidence of production releases must not be altered or deleted for a retention period, even by administrators. What should you use? A. An S3 bucket with versioning only. B. S3 Object Lock in compliance mode with a default retention. C. A CloudWatch Logs group. D. An EBS snapshot. Answer: B. Compliance mode prevents any user, including the root user, from deleting or overwriting locked versions until retention ends. Versioning alone allows deletion of versions by permitted principals.
Completion checklist
- OIDC roles exist and
trust-lintis clean on all of them -
mainis protected;cijobs are required checks -
testandprodenvironments have the protections in the table - One release reached prod by digest, with a recorded approval
- I broke readiness on purpose and watched the rollback
- I have an evidence record and can explain every field
- I destroyed the ECS, ALB and Terraform resources and confirmed nothing is billing
Key takeaways
- A pipeline is a control: it enforces who can release, proves what was released and makes rollback routine.
- Build once, promote the same digest, and record the digest.
- Identity is per job and per environment; there is nothing to steal.
- Rehearsed rollback beats designed rollback.
- Evidence is a product of the pipeline, not an afterthought, and its rules belong to your bank's policy, not this lab's.
Further reading
Independent learning resource โ not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Release management and audit evidence: from ticket to commit to deployment
Turn change approval, segregation of duties and traceability into pipeline code: a release record that links ticket, pull request, commit, image digest, approvals and scans, a release policy that blocks bad releases, and tamper-resistant evidence storage.
Module 9 cheat sheet
One-page revision summary for DevOps & CI/CD: services, Azure DevOps mapping, commands, limits and defaults, decision rules, exam traps and cost traps.