Lab 3.3: push to ECR and deploy to Fargate behind an ALB
Create a private ECR repository with immutable tags and scan on push, push an arm64 image of the Accounts API, deploy it with CDK to ECS on Fargate in private subnets behind an Application Load Balancer, test it, then destroy everything within the hour.
Assumes: .NET 10 LTS · Amazon.CDK.Lib 2.277.0 · Cdklabs.CdkNag 3.0.2 · AWS CLI v2 · Docker buildx · Region us-east-1 · labs/m03-accounts-api. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lab you can:
- Deploy a private ECR repository with immutable tags, scan on push and lifecycle rules from CDK, and read its scan findings.
- Build and push an arm64 (Graviton) image with
docker buildx. - Deploy the Accounts API to ECS on Fargate: tasks in private subnets without public IPs, an internet-facing ALB in public subnets, a task role with read-only data access, logs with retention, a deployment circuit breaker, and target tracking scaling.
- Verify the deployment, read the logs, trigger a scaling or failure scenario, and tear everything down with a tag check.
Prerequisites
- Deploying to ECS on Fargate, Health checks and Amazon ECR
- Lab 3.2 — the image builds and runs locally
- AWS CLI v2 signed in:
aws sso login --profile learn-sandbox;export AWS_PROFILE=learn-sandbox - Node.js 22 or 24 LTS and the CDK CLI (
npm install -g aws-cdk); account bootstrapped once withcdk bootstrap aws://<account>/us-east-1 - Docker Desktop with buildx (included)
Why this matters in banking
Fargate behind an ALB in private subnets is Harbourline's default shape for new APIs. Each piece is a control an auditor will ask about: images come only from a private registry and are scanned; tags cannot be overwritten (so "v1" in the change record is exactly what ran); tasks have no public IP; the app's permissions are least privilege; logs are retained for a defined time; failed deployments roll back automatically. Doing it once by hand with IaC makes every later conversation with security and audit easier.
The idea in plain language
ECR is your private Docker registry in AWS. ECS is AWS's container orchestrator: you describe a task definition (image, CPU, memory, ports, roles, logs) and a service keeps N copies running. Fargate runs those tasks without you managing servers. The ALB receives HTTP traffic and spreads it across healthy tasks.
Analogy: ECR is the bank's secure stationery store with numbered, sealed packs (immutable tags). The ECS service is the branch manager who makes sure three cashiers are always on duty, replaces any who fall ill (health checks), and calls in more at busy times (scaling). The ALB is the queue system at the door.
Architecture
Security boundary view. Only the ALB is reachable from the internet. Its security group allows port 80 inbound; the task security group allows 8080 only from the ALB's security group (the CDK pattern wires this). Tasks reach AWS APIs through the NAT gateway (and DynamoDB through a free gateway endpoint). The task execution role (pull image, write logs) is separate from the task role (read one table, read one parameter path).
Step-by-step
All commands run from labs/m03-accounts-api in Git Bash unless noted.
Deploy shared data and seed it
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
cd infra/cdk && cdk deploy HlM03-Shared -c owner=<your-name> && cd ../..
for f in seed/batch-*.json; do aws dynamodb batch-write-item --request-items "file://$f"; done
aws ssm put-parameter --name /hl-m03/accounts-api/Pseudonymisation/Key --type SecureString \
--value "$(openssl rand -hex 32)" \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m03-accounts-api Key=Environment,Value=learn Key=Owner,Value=<your-name>Each batch-write-item should return "UnprocessedItems": {}.
Create the repository
cd infra/cdk && cdk deploy HlM03-Registry -c owner=<your-name> && cd ../..
aws ecr describe-repositories --repository-names hl-m03-accounts-api \
--query "repositories[0].[imageTagMutability,imageScanningConfiguration.scanOnPush]"Expected: ["IMMUTABLE", true].
Build and push an arm64 image
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
REGISTRY=$ACCOUNT.dkr.ecr.us-east-1.amazonaws.com
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY
docker buildx build --platform linux/arm64 -t $REGISTRY/hl-m03-accounts-api:v1 --push .Try pushing v1 again after a trivial change: ECR rejects it (tag invalid: The image tag 'v1' already exists ... cannot be overwritten because the repository is immutable). That is the control working.
Read the scan results
aws ecr wait image-scan-complete --repository-name hl-m03-accounts-api --image-id imageTag=v1
aws ecr describe-image-scan-findings --repository-name hl-m03-accounts-api --image-id imageTag=v1 \
--query "imageScanFindings.findingSeverityCounts"A chiseled image typically shows few or no findings. Anything CRITICAL or HIGH would block a Harbourline release until triaged.
Deploy the Fargate stack
cd infra/cdk
cdk diff HlM03-Fargate -c owner=<your-name> -c imageTag=v1 # review IAM and security group changes
cdk deploy HlM03-Fargate -c owner=<your-name> -c imageTag=v1CDK prints the IAM and security group changes and asks for confirmation. Deployment takes about 5–8 minutes (VPC, NAT gateway, ALB, service). The output HlM03-Fargate.ApiUrl is the ALB URL. cdk-nag v3 runs at synth time; every acknowledged finding has a written reason in FargateStack.cs.
Test it
API=$(aws cloudformation describe-stacks --stack-name HlM03-Fargate \
--query "Stacks[0].Outputs[?OutputKey=='ApiUrl'].OutputValue" --output text)
curl -i $API/health/ready
curl -s $API/v1/accounts/ACC-0000000001
curl -s "$API/v1/accounts/ACC-0000000001/transactions?limit=3"
curl -i $API/v1/accounts/ACC-9999999999Note the X-Correlation-ID header: behind the ALB it is the Root= value of X-Amzn-Trace-Id (format 1-xxxxxxxx-...).
Look inside
aws logs tail /hl-m03/accounts-api/ecs --since 10m
aws ecs describe-services --cluster hl-m03-cluster --services hl-m03-accounts-api \
--query "services[0].[desiredCount,runningCount,deployments[0].rolloutState]"
aws elbv2 describe-target-health --target-group-arn $(aws elbv2 describe-target-groups \
--query "TargetGroups[?contains(TargetGroupName,'HlM03')].TargetGroupArn | [0]" --output text)Optional: watch the circuit breaker
Push a deliberately broken image as v2 (for example set ENV Repository__TableName=does-not-exist in a copy of the Dockerfile, so readiness fails), deploy with -c imageTag=v2, and watch ECS events: new tasks fail health checks, the circuit breaker trips and the service rolls back to v1 automatically. The stack update reports failure. Redeploy -c imageTag=v1 to bring CloudFormation back in sync.
Tear down (do not skip)
See Cost and cleanup below. Aim to finish within the hour.
Expected output
$ curl -i $API/health/ready
HTTP/1.1 200 OK
Content-Type: application/json
X-Correlation-ID: 1-67a3f2c1-0f1e2d3c4b5a69788796a5b4
{"status":"Healthy","checks":{"repository":"Healthy"},"durationMs":7.9}
$ aws ecs describe-services ...
[ 1, 1, "COMPLETED" ]The readiness check now takes a few milliseconds because it calls DynamoDB DescribeTable through the gateway endpoint.
What the CDK stack configures (and why)
Setting in FargateStack.cs | Purpose |
|---|---|
Vpc with MaxAzs = 2, public + PRIVATE_WITH_EGRESS subnets, NatGateways = 1 | ALB in public subnets, tasks in private ones; one NAT gateway to save cost (⚠ single-AZ egress: production uses one per AZ or VPC endpoints) |
AddFlowLog(... REJECT) with 1-week retention | Evidence of blocked traffic, at low cost |
AddGatewayEndpoint(DYNAMODB) | Free; DynamoDB traffic avoids NAT charges and stays on the AWS network |
ContainerInsightsV2 = ENABLED | Task-level CPU/memory metrics for scaling analysis (has a cost; see below) |
Task role: GetItem, Query, DescribeTable on one table + its indexes; GetParametersByPath on one path | Least privilege for the application |
LogGroup with ONE_WEEK retention, DESTROY | Never-expire logs avoided; teardown leaves nothing |
Cpu = 256, MemoryLimitMiB = 512, CpuArchitecture = ARM64 | Smallest Fargate size; Graviton is cheaper per vCPU-hour |
AssignPublicIp = false, TaskSubnets = PRIVATE_WITH_EGRESS | No direct internet exposure for tasks |
CircuitBreaker = { Enable, Rollback } | Failed deployments roll back automatically |
HealthCheckGracePeriod = 60 s, ALB check /health/ready every 15 s, 2 healthy / 3 unhealthy | Time for .NET to start; fast enough detection without flapping |
deregistration_delay = 30 s | In-flight requests drain quickly during deployments |
AutoScaleTaskCount(1..3) + CPU target 60% | Target tracking with a hard maximum |
Environment: Auth__Mode=Disabled + override | ⚠ Learning shortcut; see below |
Security considerations
⚠ Learning shortcut — not for production
The ALB listens on HTTP port 80 without TLS, has no WAF, no access logs, and the API runs with authentication disabled (explicit override). It serves fictional data and must be destroyed the same hour. Production: HTTPS listener with an ACM certificate (and redirect 80→443), AWS WAF with managed rule groups and rate limiting, ALB access logs to a locked-down S3 bucket, Auth__Mode=Cognito with issuer and client IDs, a NAT gateway per AZ or VPC interface endpoints for ECR, Logs and SSM, and enhanced (Amazon Inspector) image scanning.
- Immutable tags make the deployed artefact traceable; combine with image signing for a stronger supply-chain story.
- The task role is not the execution role. If the app gets
AccessDeniedfrom DynamoDB, fix the task role; if the task cannot pull the image or write logs, fix the execution role. - No SSH, no exec by default. Debug with logs and metrics; enable ECS Exec only temporarily and with logging if you must.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
cdk deploy fails: This stack uses assets ... bootstrap | Account/Region not bootstrapped | cdk bootstrap aws://<account>/us-east-1 |
| cdk-nag validation report fails synth | A new finding without an acknowledgment | Fix the resource, or add an Acknowledgment with the exact finding ID and a reason |
Tasks stop with CannotPullContainerError | Tag not pushed, wrong Region, or no route to ECR | aws ecr list-images; confirm the NAT gateway and private route tables |
exec format error / exit 139 | amd64 image on ARM64 tasks | Build with --platform linux/arm64 |
| Tasks start then are killed, targets "unhealthy" | /health/ready fails: table missing, wrong Region, missing parameter | Read /hl-m03/accounts-api/ecs logs; check step 1 |
App exits with Configure Pseudonymisation:Key | Parameter not created or wrong path | Create /hl-m03/accounts-api/Pseudonymisation/Key |
| 502 from the ALB right after deploy | No healthy targets yet | Wait for the grace period; check target health |
| 503 from the ALB | Zero registered targets (service scaled to 0 or all failing) | describe-services events |
Stack stuck UPDATE_IN_PROGRESS | Circuit breaker rolling back, or tasks never stabilise | Watch ECS service events; it times out or rolls back |
Cost and cleanup
Approximate us-east-1 costs while running (⚠ VERIFY current prices on the AWS pricing pages before you start):
| Resource | Approx. per hour |
|---|---|
| NAT gateway | USD 0.045 + USD 0.045 per GB processed |
| Application Load Balancer | about USD 0.0225 + LCU usage (tiny for lab traffic) |
| Public IPv4 addresses (ALB in 2 AZs + NAT) | USD 0.005 each |
| Fargate arm64, 0.25 vCPU + 0.5 GB, 1 task | about USD 0.01 |
| Container Insights, flow logs, CloudWatch Logs, DynamoDB on-demand, ECR storage | cents for a one-hour lab |
| Total | roughly USD 0.10 per hour; under USD 0.25 if destroyed within the hour |
Free alternative: Lab 3.2 locally, plus cdk synth (no deployment) to inspect the generated CloudFormation and the cdk-nag report.
cd infra/cdk
cdk destroy HlM03-Fargate HlM03-Registry HlM03-Shared -c owner=<your-name> --force
aws ssm delete-parameter --name /hl-m03/accounts-api/Pseudonymisation/Key
# Confirm nothing tagged for this lab remains (deleted resources can take a few minutes to disappear)
aws resourcegroupstaggingapi get-resources \
--tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m03-accounts-api \
--query "ResourceTagMappingList[].ResourceARN"
# Belt and braces: no NAT gateways or load balancers left
aws ec2 describe-nat-gateways --filter Name=state,Values=available --query "NatGateways[].NatGatewayId"
aws elbv2 describe-load-balancers --query "LoadBalancers[].LoadBalancerName"The registry stack uses EmptyOnDelete, so the repository is deleted with its images.
Exercises
- 🟢 Guided: Scale the service to 2 tasks with
aws ecs update-service --cluster hl-m03-cluster --service hl-m03-accounts-api --desired-count 2, call the API 20 times and confirm both tasks log requests (aws logs tailshows two stream names). - 🟡 Challenge: Change the scaling policy in
FargateStack.cstoScaleOnRequestCount(requests per target), redeploy, and generate load with a loop ofcurlcalls. Find the scaling activity inaws application-autoscaling describe-scaling-activities --service-namespace ecs. - 🔴 Live problem: After a deployment, half of the requests fail with 502 for about 20 seconds, then everything is fine. Logs show tasks being stopped while still handling requests. What settings do you investigate?
Solutions
aws logs tail /hl-m03/accounts-api/ecs --since 5m prints lines prefixed with two different stream names (api/api/<task-id-1> and api/api/<task-id-2>), and request logs appear under both. The ALB round-robins across healthy targets. Note that the auto scaling policy (min 1) may scale back in after its cooldown if CPU is low; that is expected. Set it back with --desired-count 1 or let target tracking manage it.
Replace ScaleOnCpuUtilization with scaling.ScaleOnRequestCount("Requests", new RequestCountScalingProps { RequestsPerTarget = 100, TargetGroup = service.TargetGroup }) (a low number so a curl loop can trigger it), cdk deploy, then run for i in $(seq 1 3000); do curl -s -o /dev/null $API/v1/accounts/ACC-0000000001; done. Within a few minutes describe-scaling-activities shows "Successfully set desired count to 2". Restore the CPU policy afterwards; 100 requests per minute per task is far too low for real use.
Old tasks are being stopped while the ALB still sends them requests, or new tasks receive traffic before they are ready. Check: (1) target group deregistration_delay (the stack sets 30 s; the app must keep serving during it); (2) the app handles SIGTERM gracefully and ECS stopTimeout is long enough; (3) minimumHealthyPercent is 100 so old tasks stay until new ones are healthy; (4) the readiness path really reflects readiness (not just liveness) and the health check thresholds; (5) HealthCheckGracePeriod. The fix is usually a combination of correct readiness, deregistration delay and graceful shutdown.
Knowledge check
Answer: Pushing a different image under an existing tag. It guarantees that the tag recorded in a change ticket identifies exactly one image, which makes deployments auditable and rollbacks reliable.
Answer: They have no public IPs, but must reach ECR (image pull), CloudWatch Logs and SSM over AWS public endpoints. A NAT gateway or interface endpoints provide that path; DynamoDB uses a gateway endpoint.
Answer: ECS stops the failing deployment and rolls the service back to the last completed deployment (the previous task definition), so the old version keeps serving.
Certification practice
A company runs a containerised API on Amazon ECS with AWS Fargate. Tasks must not be directly reachable from the internet, but the API must be publicly accessible over HTTPS. Which design meets these requirements?
A. Run tasks in public subnets with public IPs and restrict the security group to port 443. B. Run tasks in private subnets without public IPs behind an internet-facing ALB in public subnets; allow the task security group only from the ALB security group. C. Run tasks in private subnets and expose them through a NAT gateway. D. Use a Network Load Balancer in private subnets with public IPs on the tasks.
Answer: B. The ALB is the only internet-facing component. A exposes tasks directly. C: NAT gateways provide outbound access only. D contradicts itself and exposes tasks.
A developer deploys a new task definition revision to an ECS service. The new tasks fail health checks repeatedly. The team wants ECS to restore the previous version automatically. What should be configured?
A. ECS deployment circuit breaker with rollback enabled. B. A longer health check grace period. C. Target tracking scaling. D. ECR lifecycle policies.
Answer: A. The circuit breaker detects failed deployments and rolls back. B may help slow starters but does not roll back. C and D are unrelated to deployment failure.
Completion checklist
- ECR repository is immutable with scan on push; I read the scan findings
- The arm64 image ran on Fargate in private subnets and answered through the ALB
- I saw logs in
/hl-m03/accounts-api/ecswith correlation IDs - I destroyed all stacks, deleted the parameter and the tag check returned nothing
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab 3.2: containerise the Accounts API with a chiseled, non-root image
Build the module project's ASP.NET Core API into a multi-stage, chiseled, non-root container image, run it locally with docker compose under production-like restrictions, and check its size, user and start-up time.
Lab 3.4: run the Accounts API on Lambda (dotnet10) behind an HTTP API
Publish the same ASP.NET Core Accounts API for linux-arm64, deploy it to the Lambda dotnet10 runtime behind an API Gateway HTTP API, replay a real HTTP API event, read cold starts from the REPORT lines, and compare memory sizes, ReadyToRun and SnapStart.