Skip to Content
Cloud Engineering Academy
9. DevOps & CI/CD

Module project: complete CI/CD for an ASP.NET Core API

Build, scan and sign off a release of the Harbourline Accounts API end to end: OIDC everywhere, one image built once and deployed by digest to test and then, after a human approval and a release-policy check, to production with ECS blue/green and alarm rollback, ending in an append-only evidence pack.

advancedโฑ 4 h ๐ŸŸก Low costDOP-C02DVA-C03

Assumes: .NET 10 LTS ยท GitHub Actions (configure-aws-credentials v6.3.0, amazon-ecs-deploy-task-definition v2.6.4, trivy-action v0.36.0) ยท Amazon ECS native blue/green ยท Terraform 1.10 or later (AWS provider 6.x) ยท AWS CLI v2 ยท Region us-east-1 ยท labs/m09-delivery-pipeline. Verified against official sources on 2026-11-18.

What you are building

A delivery pipeline that a bank's internal audit would accept as a control, not just a convenience. For the Module 3 Accounts API it will:

  1. build once, test, scan and push an immutable image to Amazon ECR;
  2. deploy that exact image, referenced by digest, to a test environment;
  3. wait for a human approval on a protected prod environment, made by someone other than the author;
  4. check the release against a written release policy (ticket, digest, scans, segregation of duties);
  5. deploy to production with ECS blue/green, where a CloudWatch alarm or a failed health check rolls traffic back automatically;
  6. store a tamper-resistant evidence record for the release.

No AWS access key exists anywhere in the design. All data and names are fictional.

Learning objectives

By the end of the project you can:

  • Operate a pipeline in which each job receives short-lived credentials for one role, scoped by repository and environment.
  • Promote one image by digest through environments without rebuilding it.
  • Configure GitHub environments with required reviewers, prevent self-review and a branch restriction.
  • Rehearse automatic rollback by breaking the readiness check on purpose, and explain exactly what ECS did.
  • Produce and verify a release-evidence record that links ticket, commit, reviewers, scans, approvals and image digest.
  • Tear everything down and prove nothing is billing.

Prerequisites

Cost label

PartLabelNotes
Lab 9.1 and 9.3 workflows, trust-*, gate and evidence tools๐ŸŸข freePublic repositories: free runner minutes. Private: monthly allowance โš  VERIFY
OIDC provider, IAM roles, small ECR repository, evidence bucket๐ŸŸข free to a few centsDestroyed at the end
ECS Fargate service plus Application Load Balancer for the blue/green exercise๐ŸŸก creditsRoughly an hour of two small Fargate tasks plus an ALB: well under a dollar, but price it yourself in the AWS Pricing Calculator โš  VERIFY; destroy the same day
Lab 9.5 Lambda canary๐ŸŸข freeWithin the always-free allowance
Lab 9.6 CodeBuild and CodePipeline๐ŸŸก creditsOptional; V2 pipelines bill per action minute, with a monthly free quota โš  VERIFY

Free alternative: do everything except the Fargate deployment. The release-policy tool, health gate, trust evaluation and evidence record all run offline, and you can read the blue/green configuration as a design exercise.

Architecture

Trust boundaries

Each arrow exists because a trust policy names that exact subject. There is no arrow from ecr-push to a deploy role, and none from a fork anywhere.

Microsoft / Azure comparison

You know (Azure DevOps)This projectWhat's different
Multi-stage YAML pipeline with stagesJobs in release.yml with needsSame idea; jobs run on fresh runners
Environments with approvals and checksGitHub environments with required reviewers"Prevent self-review" is a built-in setting
Service connection to AzureOIDC role per environmentNo secret exists
Deployment slots and swapECS blue/green with test listener and bake timeTraffic is shifted by the ALB; rollback is also automatic on alarm
Pipeline artefact "drop"Image digest plus build-evidence artefactDeploy by digest, not by build number
Release gates (query monitor, work item)Release-policy check and CloudWatch alarmsWritten as code and tested

Part 1: the AWS side (Lab 9.2, Part B)

Apply the Terraform

cd labs/m09-delivery-pipeline/infra/terraform   # use your copy
cp backend.hcl.example backend.hcl
cp terraform.tfvars.example terraform.tfvars
terraform init -backend-config=backend.hcl
terraform plan -out tf.plan
terraform apply tf.plan
terraform output

You get the OIDC provider, hl-m09-gh-ecr-push, hl-m09-gh-deploy-test, hl-m09-gh-deploy-prod, the ECR repository hl-m09-accounts-api (immutable tags, scan on push, lifecycle policy) and the evidence bucket hl-m09-release-evidence-<account id> (versioning, KMS, public access blocked, TLS-only policy, Object Lock in GOVERNANCE mode for one day).

Host the service

The deploy roles may update only service/hl-m09-cluster/hl-m09-accounts-api and pass only the roles hl-m09-accounts-api-task and hl-m09-accounts-api-execution. Deploy Module 3's Fargate stack with those names, with an ALB that has a production listener and a test listener and two target groups, hl-m09-blue and hl-m09-green. Then switch the service to the native blue/green strategy using the lab's configuration:

aws ecs update-service --cli-input-json file://deploy/ecs/service-deployment.json --profile learn-sandbox

Edit the placeholder ARNs in that JSON first (target groups, listener rules and the infrastructure role). The file selects BLUE_GREEN, a 10-minute bake time, two alarms with rollback enabled (hl-m09-accounts-api-5xx and hl-m09-accounts-api-p99-latency), and a PAUSE lifecycle hook at POST_TEST_TRAFFIC_SHIFT. Create those two CloudWatch alarms before you update the service. โš  VERIFY the field names and the ALB requirements against the ECS API reference and ECS deployment strategies before applying: this feature is recent and the console may show a different layout.

โš  Learning shortcut โ€” not for production

Test and prod share one account, one cluster name and the same permissions. Production uses its own AWS account, cluster, alarms and role, and an SCP that blocks workload changes except by the pipeline. The same service-deployment.json is applied in both; only the ARNs differ.

Part 2: the GitHub side

Create the repository

Copy the contents of the lab folder to the root of a new repository (workflows must be in .github/workflows at the root). Enable the repository rules from Git and GitHub for regulated teams: a ruleset on main requiring a pull request, one approving review, code-owner review, required status checks (build-and-test, container-build, iam-policy-lint) and no force pushes. The lab's .github/CODEOWNERS names who must review pipeline, IAM and release-rule changes; replace the placeholder owners with real accounts or teams.

Create the environments

In Settings โ†’ Environments create test and prod.

Settingtestprod
Required reviewersnoneat least one person who is not a typical author
Prevent self-reviewn/aon
Deployment branchesmainmain only
VariablesAWS_DEPLOY_ROLE_ARN, ECS_CLUSTER, ECS_SERVICE, SMOKE_URLthe same four plus EVIDENCE_BUCKET, CHANGE_TICKET_PATTERN

Environment protection rules on private repositories may require a paid plan โš  VERIFY your plan. Repository-level variables: AWS_REGION, ECR_REPOSITORY, ECR_PUSH_ROLE_ARN, ECR_REGISTRY. Use the Terraform outputs for the role ARNs. Role ARNs are variables, not secrets: they identify a role, and seeing them in logs helps auditors.

Read release.yml as a design

The file's top comment is the architecture. Four decisions deserve attention.

DecisionWhereWhy
permissions: contents: read by default, id-token: write only on jobs that assume a roletop and per jobLeast privilege for GITHUB_TOKEN and OIDC
concurrency: group: release, cancel-in-progress: falsetopTwo releases never overlap, and a running deployment is never cancelled half-way
Scan before push: build with load: true, push: false, run Trivy with severity: CRITICAL, ignore-unfixed: true, exit-code: "1"build-pushA critical, fixable vulnerability never reaches the registry
deploy-prod has if: github.event_name == 'workflow_dispatch'deploy-prodPushes to main flow only as far as test. A production release is a deliberate, ticketed run with two inputs: change_ticket and version

Follow the image from commit to digest

In build-push the job: computes the tags with the lab's own image-tags tool (git-<12 hex chars> and the semantic version, never latest); assumes the ECR push role; logs in to ECR; builds and scans locally; pushes with provenance: mode=max and sbom: true; and then publishes the digest-pinned reference as the job output image. Every later job deploys repository@sha256:<digest>, so test and prod run byte-identical images. It also creates a GitHub build-provenance attestation (actions/attest-build-provenance) and an SPDX SBOM (anchore/sbom-action), and uploads the Trivy report and SBOM as the build-evidence artefact for 90 days.

Why digest and not tag: a tag is a label somebody can move; ECR immutability prevents that for you, but a digest is also verifiable by anyone downstream and is what ECS records as the running image.

Deploy to test

deploy-test declares environment: test, so the OIDC subject becomes repo:<owner>/<repo>:environment:test, which only hl-m09-gh-deploy-test trusts. It renders deploy/ecs/taskdef.json with the digest image and build metadata, registers it, updates the service and waits (up to 20 minutes) for stability. Then it checks that the running version is the commit we built:

curl -fsS --retry 10 --retry-delay 6 --retry-all-errors "$SMOKE_URL/version"
# {"version":"1.4.0","commit":"3f9c2ab1...","environment":"test","releaseId":"accounts-api-1.4.0"}

The workflow greps for "commit":"<sha>". A deployment that succeeded in ECS but is running old code fails here.

Approve and deploy to prod

Trigger a release:

GitHub โ†’ Actions โ†’ release โ†’ Run workflow, inputs: change_ticket = CHG-104233, version = 1.4.0.

The deploy-prod job stops at "Waiting for review". The reviewer sees the environment, the run and the commit. After approval the job:

  1. downloads the Trivy report and SBOM;
  2. assembles the evidence with the GitHub API: the pull request that produced the commit, who authored and merged it, who approved the PR, the run's approvals, and the Trivy critical and high counts;
  3. runs dotnet run --project src/Harbourline.Delivery -- evidence evidence-out/release.json, the release policy, which fails the job on any violation;
  4. assumes the prod role through OIDC, renders the task definition with the same digest, and updates the service. ECS performs the blue/green deployment, shifts traffic, and bakes;
  5. smoke-tests /version against production;
  6. in an if: always() step writes the evidence record to S3 with --if-none-match '*', so an existing record is never overwritten.

Part 3: the release policy, offline

The evidence command encodes the rules from the release-evidence lesson. Run it against the sample record:

dotnet run --project src/Harbourline.Delivery -- evidence samples/release-evidence.json
Release accounts-api-2026.11.18-1  sha256=2083e43c964ef975014d16f0b08f2cfc5bfc89e999ff965d70972f4c65f5df8d
Release policy satisfied.

Now break it. Change "approver": "grace.adeyemi" to "priya.shah" (the commit author) and run again:

Release accounts-api-2026.11.18-1  sha256=...
  VIOLATION: The commit author approved their own production release.
Release blocked.

The sha256 changes with the content: the record's hash is stored with the object, so a later edit is detectable. The policy rejects: no change ticket, a short SHA, an image referenced by tag, failed tests, zero tests, critical findings without a listed risk acceptance, no prod approval, a PR with no reviewer other than the author, and an author approving their own release.

Where the policy comes from

These rules are an example. Whether one person may both merge and approve a production release, how many reviewers are required, and what counts as an accepted risk are decisions for your bank's change-management policy, risk and audit teams. Never claim a pipeline makes you compliant; it produces evidence for people who decide that.

One known simplification in release.yml: the tests block of the evidence is a placeholder (passed: 1). The ๐ŸŸก exercise below replaces it with the real counts from the test-results artefact.

Part 4: blue/green, and breaking it on purpose

Rehearse the failure

The API exposes /health/live and /health/ready. Readiness reads the setting Health__ForceUnready; when it is true, readiness returns 503 while liveness stays 200 (liveness must not fail or ECS would restart-loop the task).

  1. Locally, prove the behaviour:

    docker run --rm -p 8080:8080 -e Health__ForceUnready=true hl-m09-accounts-api:dev
    curl -i localhost:8080/health/ready    # 503
    curl -i localhost:8080/health/live     # 200
  2. In your repository change the Health__ForceUnready value in deploy/ecs/taskdef.json to "true" on a branch for the test environment only, merge by pull request, and run the release. In ECS you will see the green tasks start, the target group mark them unhealthy, and the deployment fail without production traffic ever reaching the bad version. For prod, the same change would be caught by the alarms and the bake time. In both cases blue keeps serving.

  3. Revert the setting by a new pull request and a new version. Because tags are immutable, you cannot re-push 1.4.0; releasing 1.4.1 is the honest, auditable path.

  4. Read the history: Console โ†’ ECS โ†’ Clusters โ†’ your cluster โ†’ Services โ†’ Deployments, or:

    aws ecs list-service-deployments --cluster hl-m09-cluster --service hl-m09-accounts-api --profile learn-sandbox

    โš  VERIFY the CLI command names in your CLI version; the service-deployment APIs are recent.

The decision logic of a rollback gate is also available offline as the gate tool, which turns health samples into Proceed, Hold or Rollback:

dotnet run --project src/Harbourline.Delivery -- gate samples/canary-failing.json
Decision: Rollback
  - Error rate 10.30 % exceeds 1.00 %.
  - p99 latency 1020 ms exceeds 800 ms.

and samples/canary-healthy.json gives Decision: Proceed with Error rate 0.00 %, p99 295 ms โ€” within policy. Thresholds (1% errors, 800 ms p99, at least 50 requests for a verdict, two consecutive bad samples) are written down deliberately: they are the objective go or no-go criteria a change board wants to read. See Rollback strategies for why database changes can make "roll back" unsafe.

Lab 9.3: scanning and SBOM (security.yml)

The security workflow runs on pull requests, on main and weekly (Mondays 06:17 UTC), because new vulnerabilities appear for code that has not changed.

JobToolGate
secretsGitleaks over the full history (fetch-depth: 0)Any secret fails
dependenciesNuGet Audit (restore) plus dotnet package list --vulnerable --include-transitiveHigh and critical fail the restore
imagedocker build, SPDX SBOM (Anchore), Trivy table report, Trivy JSON gateFixable CRITICAL fails

Trivy is pinned to v0.36.0 by SHA, with a comment in the workflow: the lab never uses a trivy-action tag older than v0.35.0 because of a compromise of that action in March 2026 โš  VERIFY the incident details in the project's advisory before citing them. This is the practical lesson in why you pin actions. Prove the gate: add an obviously fake AWS-style key to a file on a branch and watch the secrets job fail, then remove it and treat the key as burned (history keeps it). Use only the documented example key from AWS docs, never a real one. For tool depth see Pipeline security scanning in Module 5 and Pipeline security.

Lab 9.5: Lambda canary (design and CLI)

deploy/lambda/appspec.yml is the raw form of what AWS SAM generates for you. It names the function, the alias live, the current and target versions, and two hooks (BeforeAllowTraffic, AfterAllowTraffic). The deployment group uses the configuration CodeDeployDefault.LambdaCanary10Percent5Minutes (10% of traffic for five minutes, then the rest) and an alarm on hl-m09-statements-errors.

# After publishing version 4 of the function and creating the deployment group:
aws deploy create-deployment \
  --application-name hl-m09-statements \
  --deployment-group-name hl-m09-statements-canary \
  --revision '{"revisionType":"AppSpecContent","appSpecContent":{"content":"<appspec as a JSON string>"}}' \
  --profile learn-sandbox

โš  VERIFY the exact --revision structure for Lambda in the CodeDeploy CLI reference; most teams let SAM or CDK generate and run this. The full walkthrough, hooks and rollback behaviour are in Lambda deployments. To rehearse: make version 4 return errors and invoke the alias in a loop. The alarm trips during the five canary minutes and CodeDeploy moves the alias back to version 3. Cost is free within the always-free tier; delete the function, alias, alarm and deployment group afterwards.

Lab 9.6: the same build in CodeBuild (optional)

buildspec.yml repeats the pipeline's build in AWS CodeBuild so you can compare the two models. It runs the tests inside the official mcr.microsoft.com/dotnet/sdk:10.0 container (so results do not depend on which .NET the CodeBuild image ships), computes tags with the same image-tags tool, builds and pushes, resolves the digest, exports IMAGE_URI, writes imagedefinitions.json for the CodePipeline ECS deploy action, and publishes JUnit reports.

Requirements: the standard Amazon Linux image (the file says aws/codebuild/amazonlinux-x86_64-standard:5.0 or newer โš  VERIFY the current image name), privileged mode enabled for Docker builds, a service role limited to ECR push for one repository plus its own logs and report groups, and plain-text variables ECR_REGISTRY, ECR_REPOSITORY and APP_VERSION. In CodePipeline V2: Source (GitHub through CodeConnections โš  VERIFY current naming) โ†’ Build (CodeBuild) โ†’ Manual approval โ†’ Deploy (Amazon ECS). Differences to note: the approval lives in CodePipeline and is recorded there; identity is the service role, not OIDC; and cost is per action minute. See CodeBuild, CodePipeline and CodeDeploy.

Security considerations

โš  Learning shortcut โ€” not for production

GOVERNANCE-mode Object Lock for one day lets an administrator with the bypass permission delete evidence, and force_destroy removes the bucket in the lab. A bank sets COMPLIANCE mode with a retention period from its records policy, restricts who can change the bucket policy, and replicates evidence to a separate log-archive account.

  • No long-lived credentials. No IAM user, no key, no secret holding AWS access. If you find one, delete it and rotate.
  • Least privilege per job. The ECR role pushes to one repository; deploy roles update one service and pass two roles to ECS; only prod writes under releases/* in the evidence bucket.
  • Segregation of duties is enforced twice. By GitHub (prevent self-review, required reviewers) and by the release policy (author is not the approver).
  • Untrusted code never gets a role. Pull-request workflows have no id-token permission.
  • Dependencies and actions are pinned by SHA and updated by reviewed Dependabot PRs.
  • Evidence is append-only. If-None-Match: * refuses overwrite, versioning keeps history, Object Lock prevents deletion for the retention period, and each record carries a SHA-256.
  • Runtime secrets bypass the pipeline. The service reads its own secrets through its task roles; see Secrets in CI/CD.

Common mistakes

  • Rebuilding the image for prod "to be safe". That destroys the claim "what we tested is what we shipped".
  • Approving a production deployment on mobile without reading the commit and ticket.
  • Alarms with Missing data treated as healthy, so a dead service looks fine โš  choose the treat-missing-data setting deliberately.
  • Setting a bake time shorter than the metric period of the alarms, so an alarm cannot fire before the old tasks are gone.
  • Re-pushing a tag after a failed release. Immutable tags forbid it: bump the version.
  • Storing evidence only as workflow logs. Logs expire and are editable by admins; export the record.

Troubleshooting

SymptomLikely causeFix
build-push fails at "Assume the ECR push role"Job is not on main, so sub does not match the ECR trust policyRun from main, or check the trust policy subject
Trivy step fails the buildA fixable CRITICAL vulnerabilityUpdate the base image or package; do not add an ignore without a recorded risk acceptance
tag invalid: The image tag already existsImmutable tags and a repeated versionUse a new version
deploy-test hangs then times outNew tasks fail the target group health checkCheck /health/ready, security groups, task logs in /ecs/hl-m09-accounts-api
deploy-prod never startsWaiting for approval, or workflow_dispatch was not usedApprove in the run page, or start the workflow manually
Release policy fails with "No tests ran"The placeholder or empty test summaryFill in real counts; a green build with zero tests is not evidence
Evidence step fails with PreconditionFailedA record for that release and status already existsIntended: bump the version, or investigate why a duplicate was attempted
ECS rolled back and the workflow is redAlarm or health failure during the deploymentRead the service deployment events; fix forward with a new version
iam:PassRole deniedThe task role names in the policy differ from your stackAlign the names in oidc.tf and the Module 3 stack

Cost and cleanup

Destroy the ECS and ALB resources the same day, then the lab Terraform.

# 1. Module 3 stack (ECS service, ALB, alarms) โ€” from labs/m03-accounts-api/infra/cdk
cdk destroy --profile learn-sandbox

# 2. This lab's IAM, ECR and evidence bucket
cd labs/m09-delivery-pipeline/infra/terraform
terraform destroy
# Evidence objects are locked for 1 day (GOVERNANCE). If destroy reports locked versions, wait a day
# or remove them with --bypass-governance-retention (needs s3:BypassGovernanceRetention).

# 3. Confirm nothing from the lab remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m09-delivery-pipeline --profile learn-sandbox --region us-east-1
aws iam list-roles --query "Roles[?starts_with(RoleName, 'hl-m09-')].RoleName" --profile learn-sandbox

Also delete the CloudWatch log group /ecs/hl-m09-accounts-api, any Lambda function, CodeDeploy application and CodeBuild or CodePipeline resources from Labs 9.5 and 9.6, and the environment variables in GitHub, so a future run fails fast instead of half-way.

Exercises

  • ๐ŸŸข Guided: run the release workflow to test only (a push to main), then a manual release with a ticket. Download the evidence JSON from S3 and run the evidence tool against it.
  • ๐ŸŸก Challenge: replace the placeholder test counts. In deploy-prod, download the test-results artefact of the ci run for the same commit (via gh run download or the artefacts API), count passed, failed and skipped from the JUnit XML with jq or a small .NET tool, and put them in the evidence. Add a failing-test fixture to the unit tests of the policy.
  • ๐Ÿ”ด Live problem: auditors ask, "Show me every production change in November, who approved it, and prove the image that ran is the one that was tested." You have 30 minutes. Describe which records you pull, from where, and what you cannot prove today. Then list the three improvements you would make first.

Assignment

Publish your repository (public, or private with screenshots) containing: the workflows, the policies and the Terraform; a RELEASE-EVIDENCE.md that explains the evidence record field by field; one real evidence JSON from your run; and a short design note answering "what would change in a real bank?" (separate accounts, SCPs, COMPLIANCE-mode retention, change-management integration, who approves).

Solutions

Knowledge check

Interview questions

  • Walk me through a production release in your pipeline. Merge to protected main after review; build once; scan; push by digest; deploy to test; approval on the prod environment by someone other than the author; release-policy check; blue/green deploy with alarm rollback; evidence stored immutably.
  • How do you prove the image in production is the one you tested? The digest in the evidence record, the task definition and ECR match; plus the provenance attestation and SBOM tied to that digest.
  • A deploy failed and rolled back. What do you do next? Confirm the service is healthy on the previous version, capture the failure evidence, fix forward with a new version through the normal pipeline, and record the incident. Never re-push an existing tag.

Certification practice

Completion checklist

  • OIDC roles exist and trust-lint is clean on all of them
  • main is protected; ci jobs are required checks
  • test and prod environments have the protections in the table
  • One release reached prod by digest, with a recorded approval
  • I broke readiness on purpose and watched the rollback
  • I have an evidence record and can explain every field
  • I destroyed the ECS, ALB and Terraform resources and confirmed nothing is billing

Key takeaways

  • A pipeline is a control: it enforces who can release, proves what was released and makes rollback routine.
  • Build once, promote the same digest, and record the digest.
  • Identity is per job and per environment; there is nothing to steal.
  • Rehearsed rollback beats designed rollback.
  • Evidence is a product of the pipeline, not an afterthought, and its rules belong to your bank's policy, not this lab's.

Further reading

Independent learning resource โ€” not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page