Deploying .NET to ECS on Fargate
Deploy a containerised ASP.NET Core API to Amazon ECS on Fargate in private subnets behind an Application Load Balancer, with separate task and execution roles, safe rolling deployments and a C# rollout watcher.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.ECS 4.1.1.1) · AWS CLI v2 · Amazon.CDK.Lib 2.277.0 · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the four ECS building blocks — cluster, task definition, task, service — and what Fargate removes from your to-do list.
- Write a Fargate task definition for an ASP.NET Core API: CPU and memory sizing, ARM64,
awsvpcnetworking, logging, secrets and a read-only root file system. - Tell the task execution role from the task role, and give each one least-privilege permissions.
- Put tasks in private subnets behind an Application Load Balancer using an
iptarget group. - Configure rolling deployments with the deployment circuit breaker so a bad release rolls itself back.
- Deploy with the AWS CLI, AWS CDK (C#) or Terraform, watch the rollout from C#, and tear everything down.
Prerequisites
- Containerising .NET and Amazon ECR: you need an image in an ECR repository.
- VPC fundamentals: public vs private subnets, route tables, NAT gateways.
- IAM in depth: roles, trust policies and
iam:PassRole. - Tools: AWS CLI v2 with the
learn-sandboxSSO profile, Docker, .NET 10 SDK. Optional: AWS CDK CLI or Terraform 1.14.
Why this matters in banking
Harbourline Bank's Accounts API answers "what is my balance?" for the mobile app. It runs all day, every day, with predictable traffic peaks at 08:00 and on payday. Three things go wrong at banks that deploy containers carelessly:
- Over-permissioned containers. A team gives the container the same role ECS uses to pull images and write logs, then adds
secretsmanager:*"to make it work". Now any remote-code-execution bug in a JSON library can read every secret in the account. - Public tasks. Someone sets
assignPublicIptoENABLEDbecause the image would not pull. The API is now reachable directly from the internet, bypassing the load balancer, the WAF and the TLS policy. - Deployments that never finish. A release with a broken health endpoint keeps starting and killing tasks for hours. Customers see intermittent 503 errors; on-call engineers see nothing obvious because no alarm watches the deployment.
An auditor will ask: Which identity can read customer data? Is the workload reachable from the internet? How do you know a release succeeded, and how do you roll back? This lesson gives you concrete answers to all three.
The idea in plain language
Amazon ECS (Elastic Container Service) is AWS's own container orchestrator. You tell it "keep two copies of this container running, register them with this load balancer, and replace any that die". ECS does the bookkeeping.
AWS Fargate is a way of running ECS tasks without managing servers. You do not create EC2 instances, patch them or size a cluster. You say "this task needs 0.25 vCPU and 512 MB", and AWS finds the capacity, runs the task in its own isolated micro-VM, and bills you per second for what you asked for.
Four words to learn:
| Term | Plain meaning | .NET analogy |
|---|---|---|
| Cluster | A logical grouping of services and tasks. With Fargate it holds no servers, just a name and settings. | A solution folder |
| Task definition | A versioned JSON recipe: which image, how much CPU and memory, which ports, environment variables, roles and log settings. Every change creates a new revision (family:1, family:2 …). | A .csproj plus launchSettings.json, frozen and versioned |
| Task | One running copy of a task definition: one or more containers sharing a network interface. | One running dotnet process |
| Service | A controller that keeps N tasks running, replaces failed ones, registers them with a load balancer and rolls out new revisions. | IIS application pool with automatic recycling, plus a deployment tool |
Analogy: Think of a bank branch staffing rota. The task definition is the job description (skills, desk, keycard level). A task is one person on shift. The service is the branch manager who makes sure there are always two cashiers at the counter, replaces anyone who goes home sick, and phases in new staff one at a time when the job description changes. Fargate is the staffing agency: you never hire, house or train anybody yourself; you just request "two cashiers, standard skills" and pay by the hour.
What problem ECS on Fargate solves
Without an orchestrator, running containers in production means writing your own scripts to start them, restart them, spread them across Availability Zones, update them without downtime and wire them to a load balancer. ECS does all of that. Fargate additionally removes the servers underneath, which matters to a bank because every EC2 instance is something you must patch, harden, scan and evidence to auditors.
What you still own: the image (and its vulnerabilities), the task definition, IAM roles, network placement, security groups, scaling rules, logs and alarms.
Limitations and sharp edges (checked 2026-11-18):
- Fixed CPU/memory combinations. For example, 0.25 vCPU (
256CPU units) allows 512 MiB, 1 GB or 2 GB of memory. Larger tasks go up to 16 vCPU and 120 GB ⚠ VERIFY the current upper limits in the task-size table before sizing large jobs. - Per-second billing with a one-minute minimum, priced per vCPU-hour and GB-hour. In us-east-1, Linux/x86 costs about USD 0.0405 per vCPU-hour and USD 0.0044 per GB-hour; Linux/ARM (Graviton) about USD 0.0324 and USD 0.0036. ARM is roughly 20% cheaper.
- 20 GB of ephemeral storage is included per task; more is charged per GB-hour.
- No host access. You cannot SSH into a Fargate task. For debugging there is ECS Exec (Session Manager into the container), but a chiseled .NET image has no shell, so in practice you debug through logs and metrics.
- No GPUs on Fargate. Use ECS on EC2 for GPU work.
- Image pull time counts towards start-up. Small images (chiseled, trimmed) start faster and scale out faster.
Where Express Mode fits. In late 2025 AWS launched Amazon ECS Express Mode. You give it an image, an execution role and an infrastructure role, and it creates the Fargate service, an HTTPS ALB with a certificate, auto scaling, alarms and a URL like servicename.ecs.region.on.aws. There is no extra charge beyond those resources. It is the official replacement for App Runner, which is closed to new customers. This lesson builds everything by hand first, because you cannot operate (or defend to an auditor) what you do not understand. The final exercise uses Express Mode so you can compare.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Container Apps | ECS on Fargate (or ECS Express Mode for the "just give me a URL" experience) | ECS exposes every piece (target groups, security groups, roles); Container Apps hides more |
| AKS | Amazon EKS | ECS is not Kubernetes: no YAML manifests, no kubectl, far fewer moving parts |
| Container App revision | Task definition revision | Revisions are immutable; a service points at exactly one revision at a time (outside blue/green) |
| Managed identity on a container app | Task role | The SDK finds the task role credentials automatically through the container credentials endpoint |
| ACR pull permission via managed identity | Task execution role | A separate role used by the ECS agent, not by your code |
| Application Gateway / Front Door | Application Load Balancer (+ CloudFront and WAF) | The ALB lives in your VPC; tasks register with a target group by IP address |
| Key Vault references in app settings | secrets in the container definition | Values are injected at task start only; rotation needs a new deployment or in-app fetching |
| Log Analytics via Container Apps environment | awslogs log driver → CloudWatch Logs | You create the log group and set retention yourself |
| Deployment slots / revision traffic split | Rolling update with circuit breaker, or blue/green deployments | ECS now has built-in blue/green; CodeDeploy is no longer required for it ⚠ VERIFY |
Architecture
Security-boundary view. Each arrow crosses a control:
The key design decisions:
- ALB in public subnets, tasks in private subnets with
assignPublicIp = DISABLED. The only way in is through the ALB. - Security-group referencing:
api-sgallows port 8080 only fromalb-sg(a security group ID), never from a CIDR. - Target type
ip. With theawsvpcnetwork mode each task gets its own elastic network interface (ENI) and private IP, so the target group registers IPs, not instances. - Outbound path. Tasks need to reach ECR, CloudWatch Logs and Systems Manager. Either a NAT gateway (simple, about USD 0.045 per hour plus data) or interface VPC endpoints (private, about USD 0.01 per endpoint per AZ per hour ⚠ VERIFY). A DynamoDB or S3 gateway endpoint is free.
The two roles — the most important idea in this lesson
| Task execution role | Task role | |
|---|---|---|
| Used by | The ECS/Fargate agent, before and around your container | Your code, through the AWS SDK |
| Typical permissions | ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer, logs:CreateLogStream, logs:PutLogEvents; plus secretsmanager:GetSecretValue / ssm:GetParameters / kms:Decrypt only for secrets referenced in the task definition | Whatever the application needs: e.g. dynamodb:GetItem and dynamodb:Query on one table |
| Trust policy principal | ecs-tasks.amazonaws.com | ecs-tasks.amazonaws.com |
| Visible to the container? | No | Yes, via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI |
| Symptom if missing | Task stops before starting: CannotPullContainerError, ResourceInitializationError | Task runs, but SDK calls fail with AccessDeniedException |
The AWS managed policy AmazonECSTaskExecutionRolePolicy covers image pulls and log writing. Never attach your application permissions to the execution role, and never attach AmazonECSTaskExecutionRolePolicy to the task role.
Step-by-step implementation
Prepare the image and the network
Push the Accounts API image to ECR with an immutable tag such as v1 (see the ECR lesson). Use a VPC with two public and two private subnets across two AZs. The lab stack in labs/m03-accounts-api/infra/cdk creates exactly this.
Create the log group with a retention period
Create /hl-m03/accounts-api/ecs with 7-day retention. If the awslogs driver creates the group for you (awslogs-create-group), it has no expiry and the execution role needs logs:CreateLogGroup. Create it yourself.
Create the two roles
Execution role with AmazonECSTaskExecutionRolePolicy. Task role with an inline policy scoped to one DynamoDB table and one Parameter Store path. Both trust ecs-tasks.amazonaws.com, ideally with an aws:SourceAccount condition against the confused-deputy problem.
Register the task definition
Choose 256 CPU units and 512 MiB, ARM64, awsvpc, port 8080, read-only root file system with a writable /tmp volume, the log configuration and non-secret environment variables.
Create the cluster, security groups, ALB and target group
Target group: protocol HTTP, port 8080, target type ip, health check path /health/ready, deregistration delay 30 seconds (the default of 300 seconds makes every deployment five minutes slower).
Create the service
Desired count 2, launch type Fargate, private subnets, no public IP, the target group, a health check grace period, the deployment circuit breaker with rollback, minimumHealthyPercent 100 and maximumPercent 200.
Wait, test, then deploy a second revision
aws ecs wait services-stable, call the ALB URL, then register revision 2 with a changed environment variable and update the service. Watch the rollout with the C# watcher below.
Tear down
Delete the service, ALB, target group, cluster, log group, roles and (if you made one) the NAT gateway and VPC. Check nothing tagged Project=cea-lab remains.
AWS Console instructions
- Console → Elastic Container Service → Clusters → Create cluster. Name
hl-m03-cluster, infrastructure AWS Fargate (serverless) only. Turn on Container Insights with enhanced observability if you want task-level metrics (it costs CloudWatch metric charges; skip it for a one-hour lab). - Console → Elastic Container Service → Task definitions → Create new task definition. Launch type AWS Fargate, OS/architecture Linux/ARM64, CPU .25 vCPU, memory .5 GB. Choose your Task role and Task execution role. Under Container – 1: name
api, image URI from ECR, container port 8080, protocol HTTP, app protocol HTTP. Tick Read only root file system. Under Log collection, choose Amazon CloudWatch and the log group you created. Add environment variables. Add a bind mount volume namedtmpand mount it at/tmp. - Console → EC2 → Target Groups → Create target group. Type IP addresses, protocol HTTP, port 8080, your VPC, health check path
/health/ready. Skip registering targets; ECS does that. - Console → EC2 → Load Balancers → Create → Application Load Balancer. Internet-facing, two public subnets, security group
alb-sg, listener forwarding to the target group. (Production: HTTPS 443 with an ACM certificate and a WAF web ACL.) - Console → Elastic Container Service → Clusters → hl-m03-cluster → Services → Create. Compute Launch type: FARGATE, task definition family and revision, desired tasks 2. Networking: your VPC, the two private subnets, security group
api-sg, Public IP turned off. Load balancing: existing ALB, existing target group, containerapi:8080. Deployment options: rolling update, min 100%, max 200%, Deployment circuit breaker: Use the Amazon ECS deployment circuit breaker with Rollback on failures ticked. Health check grace period 60 seconds. - Open Services → hl-m03-accounts-api → Deployments and Events to watch tasks start and register.
AWS CLI
The commands below assume you already have a VPC, two public subnets, two private subnets, alb-sg and api-sg. Replace the subnet-…, sg-… and account values.
# Common variables (Git Bash). PROFILE and REGION follow CONVENTIONS.md.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
TAGS='key=Project,value=cea-lab key=Lab,value=m03-fargate key=Environment,value=learn key=Owner,value=you'
# 1. Log group with retention (never infinite).
aws logs create-log-group --log-group-name /hl-m03/accounts-api/ecs \
--tags Project=cea-lab,Lab=m03-fargate,Environment=learn,Owner=you
aws logs put-retention-policy --log-group-name /hl-m03/accounts-api/ecs --retention-in-days 7
# 2. Trust policy shared by both roles. aws:SourceAccount limits who can make ECS assume it.
cat > ecs-trust.json <<EOF
{ "Version": "2012-10-17",
"Statement": [{ "Effect": "Allow",
"Principal": { "Service": "ecs-tasks.amazonaws.com" },
"Action": "sts:AssumeRole",
"Condition": { "StringEquals": { "aws:SourceAccount": "$ACCOUNT" } } }] }
EOF
# 3. Execution role: AWS managed policy for ECR pulls and log writes.
aws iam create-role --role-name hl-m03-accounts-api-exec \
--assume-role-policy-document file://ecs-trust.json
aws iam attach-role-policy --role-name hl-m03-accounts-api-exec \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
# 4. Task role: only what the application needs.
aws iam create-role --role-name hl-m03-accounts-api-task \
--assume-role-policy-document file://ecs-trust.json
cat > task-policy.json <<EOF
{ "Version": "2012-10-17",
"Statement": [
{ "Sid": "ReadAccountsTable", "Effect": "Allow",
"Action": ["dynamodb:GetItem", "dynamodb:Query", "dynamodb:DescribeTable"],
"Resource": ["arn:aws:dynamodb:us-east-1:$ACCOUNT:table/hl-m03-accounts",
"arn:aws:dynamodb:us-east-1:$ACCOUNT:table/hl-m03-accounts/index/*"] },
{ "Sid": "ReadOwnParameters", "Effect": "Allow",
"Action": ["ssm:GetParametersByPath"],
"Resource": "arn:aws:ssm:us-east-1:$ACCOUNT:parameter/hl-m03/accounts-api" } ] }
EOF
aws iam put-role-policy --role-name hl-m03-accounts-api-task \
--policy-name accounts-api-runtime --policy-document file://task-policy.json
# 5. Register the task definition (file shown in "Complete code").
aws ecs register-task-definition --cli-input-json file://task-definition.json
# 6. Cluster.
aws ecs create-cluster --cluster-name hl-m03-cluster --tags $TAGS
# 7. Target group (type ip) and listener. Deregistration delay 30 s instead of 300 s.
TG_ARN=$(aws elbv2 create-target-group --name hl-m03-api-tg --protocol HTTP --port 8080 \
--target-type ip --vpc-id vpc-0123456789abcdef0 \
--health-check-path /health/ready --health-check-interval-seconds 15 \
--healthy-threshold-count 2 --unhealthy-threshold-count 3 --matcher HttpCode=200 \
--query 'TargetGroups[0].TargetGroupArn' --output text)
aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" \
--attributes Key=deregistration_delay.timeout_seconds,Value=30
ALB_ARN=$(aws elbv2 create-load-balancer --name hl-m03-api-alb --type application \
--scheme internet-facing --subnets subnet-pub-a subnet-pub-b --security-groups sg-alb \
--query 'LoadBalancers[0].LoadBalancerArn' --output text)
# ⚠ Learning shortcut: HTTP 80. Production uses HTTPS 443 with an ACM certificate.
aws elbv2 create-listener --load-balancer-arn "$ALB_ARN" --protocol HTTP --port 80 \
--default-actions Type=forward,TargetGroupArn="$TG_ARN"
# 8. Service: private subnets, no public IP, circuit breaker with rollback.
aws ecs create-service --cluster hl-m03-cluster --service-name hl-m03-accounts-api \
--task-definition hl-m03-accounts-api --desired-count 2 --launch-type FARGATE \
--platform-version LATEST \
--network-configuration 'awsvpcConfiguration={subnets=[subnet-priv-a,subnet-priv-b],securityGroups=[sg-api],assignPublicIp=DISABLED}' \
--load-balancers "targetGroupArn=$TG_ARN,containerName=api,containerPort=8080" \
--health-check-grace-period-seconds 60 \
--deployment-configuration 'deploymentCircuitBreaker={enable=true,rollback=true},minimumHealthyPercent=100,maximumPercent=200' \
--enable-ecs-managed-tags --propagate-tags SERVICE --tags $TAGS
# 9. Block until the service reaches a steady state (polls every 15 s, gives up after 40 tries).
aws ecs wait services-stable --cluster hl-m03-cluster --services hl-m03-accounts-api
# 10. Test through the ALB.
DNS=$(aws elbv2 describe-load-balancers --load-balancer-arns "$ALB_ARN" \
--query 'LoadBalancers[0].DNSName' --output text)
curl -s "http://$DNS/health/ready"
# 11. Roll out a new revision (after re-registering the task definition with a change).
aws ecs update-service --cluster hl-m03-cluster --service hl-m03-accounts-api \
--task-definition hl-m03-accounts-api
# 12. See why tasks stopped (the single most useful debugging command).
aws ecs list-tasks --cluster hl-m03-cluster --desired-status STOPPED --query 'taskArns' --output text \
| xargs -n 10 aws ecs describe-tasks --cluster hl-m03-cluster \
--query 'tasks[].{task:taskArn,reason:stoppedReason,exit:containers[0].exitCode}' --tasksInfrastructure as Code
The lab project labs/m03-accounts-api/infra/cdk/Harbourline.Accounts.Infra/FargateStack.cs contains the full stack (VPC, flow logs, DynamoDB gateway endpoint, cluster, task role, ALB, health check, auto scaling and cdk-nag). The essentials look like this:
using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.ECR;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.ECS.Patterns;
using Amazon.CDK.AWS.IAM;
using Amazon.CDK.AWS.Logs;
using Constructs;
public sealed class AccountsFargateStack : Stack
{
public AccountsFargateStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var vpc = new Vpc(this, "Vpc", new VpcProps { MaxAzs = 2, NatGateways = 1 });
var cluster = new Cluster(this, "Cluster", new ClusterProps { Vpc = vpc, ClusterName = "hl-m03-cluster" });
// Task role: what the CODE may do.
var taskRole = new Role(this, "TaskRole", new RoleProps
{
AssumedBy = new ServicePrincipal("ecs-tasks.amazonaws.com"),
});
taskRole.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
Actions = ["dynamodb:GetItem", "dynamodb:Query", "dynamodb:DescribeTable"],
Resources = [$"arn:aws:dynamodb:{Region}:{Account}:table/hl-m03-accounts"],
}));
var logGroup = new LogGroup(this, "Logs", new LogGroupProps
{
LogGroupName = "/hl-m03/accounts-api/ecs",
Retention = RetentionDays.ONE_WEEK,
RemovalPolicy = RemovalPolicy.DESTROY,
});
var service = new ApplicationLoadBalancedFargateService(this, "Api", new ApplicationLoadBalancedFargateServiceProps
{
Cluster = cluster,
Cpu = 256,
MemoryLimitMiB = 512,
DesiredCount = 2,
RuntimePlatform = new RuntimePlatform
{
CpuArchitecture = CpuArchitecture.ARM64,
OperatingSystemFamily = OperatingSystemFamily.LINUX,
},
TaskImageOptions = new ApplicationLoadBalancedTaskImageOptions
{
Image = ContainerImage.FromEcrRepository(
Repository.FromRepositoryName(this, "Repo", "hl-m03-accounts-api"), "v1"),
ContainerPort = 8080,
TaskRole = taskRole, // the execution role is generated by CDK with minimal pull/log rights
LogDriver = LogDrivers.AwsLogs(new AwsLogDriverProps { StreamPrefix = "api", LogGroup = logGroup }),
Environment = new Dictionary<string, string> { ["ASPNETCORE_ENVIRONMENT"] = "Production" },
},
TaskSubnets = new SubnetSelection { SubnetType = SubnetType.PRIVATE_WITH_EGRESS },
AssignPublicIp = false,
CircuitBreaker = new DeploymentCircuitBreaker { Enable = true, Rollback = true },
HealthCheckGracePeriod = Duration.Seconds(60),
MinHealthyPercent = 100,
MaxHealthyPercent = 200,
});
service.TargetGroup.ConfigureHealthCheck(new Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck
{
Path = "/health/ready",
Interval = Duration.Seconds(15),
HealthyThresholdCount = 2,
UnhealthyThresholdCount = 3,
});
service.TargetGroup.SetAttribute("deregistration_delay.timeout_seconds", "30");
}
}ApplicationLoadBalancedFargateService is an L3 pattern: one construct creates the ALB, listener, target group, security groups (with the ALB-to-task rule), execution role, task definition and service.
Complete code
The task definition
{
"family": "hl-m03-accounts-api",
"requiresCompatibilities": ["FARGATE"],
"networkMode": "awsvpc",
"cpu": "256",
"memory": "512",
"runtimePlatform": { "cpuArchitecture": "ARM64", "operatingSystemFamily": "LINUX" },
"executionRoleArn": "arn:aws:iam::111122223333:role/hl-m03-accounts-api-exec",
"taskRoleArn": "arn:aws:iam::111122223333:role/hl-m03-accounts-api-task",
"volumes": [{ "name": "tmp" }],
"containerDefinitions": [
{
"name": "api",
"image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/hl-m03-accounts-api:v1",
"essential": true,
"readonlyRootFilesystem": true,
"portMappings": [{ "containerPort": 8080, "protocol": "tcp", "name": "http", "appProtocol": "http" }],
"mountPoints": [{ "sourceVolume": "tmp", "containerPath": "/tmp", "readOnly": false }],
"environment": [
{ "name": "ASPNETCORE_ENVIRONMENT", "value": "Production" },
{ "name": "Repository__Provider", "value": "DynamoDb" },
{ "name": "Repository__TableName", "value": "hl-m03-accounts" },
{ "name": "ParameterStore__Path", "value": "/hl-m03/accounts-api/" },
{ "name": "AWS__Region", "value": "us-east-1" }
],
"stopTimeout": 30,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/hl-m03/accounts-api/ecs",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "api",
"mode": "non-blocking",
"max-buffer-size": "4m"
}
}
}
],
"tags": [
{ "key": "Project", "value": "cea-lab" },
{ "key": "Lab", "value": "m03-fargate" },
{ "key": "Environment", "value": "learn" },
{ "key": "Owner", "value": "you" }
]
}A C# rollout watcher
Pipelines often fire update-service and walk away. This small console app waits for the new deployment to complete or fail, prints the service events, and returns a non-zero exit code on failure, so a GitHub Actions job goes red when ECS rolls back.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.Deploy.Watcher</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.ECS" Version="4.1.1.1" />
<PackageReference Include="AWSSDK.SSO" Version="4.1.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.1.0.2" />
</ItemGroup>
</Project>using Amazon.ECS;
using Amazon.ECS.Model;
using Task = System.Threading.Tasks.Task; // Amazon.ECS.Model also defines a Task type (an ECS task)
// Usage: dotnet run -- <cluster> <service> [timeoutMinutes]
if (args.Length < 2)
{
Console.Error.WriteLine("Usage: Harbourline.Deploy.Watcher <cluster> <service> [timeoutMinutes]");
return 2;
}
var cluster = args[0];
var serviceName = args[1];
var timeout = TimeSpan.FromMinutes(args.Length > 2 && int.TryParse(args[2], out var m) ? m : 15);
// Region and credentials come from the default chain: AWS_PROFILE=learn-sandbox locally,
// the runner's OIDC role in CI. No keys in code.
using var ecs = new AmazonECSClient();
using var cts = new CancellationTokenSource(timeout);
var seenEvents = new HashSet<string>();
try
{
while (true)
{
var response = await ecs.DescribeServicesAsync(new DescribeServicesRequest
{
Cluster = cluster,
Services = [serviceName],
}, cts.Token);
// SDK v4: collections can be null and value types are nullable. Guard every access.
var service = response.Services?.FirstOrDefault()
?? throw new InvalidOperationException($"Service {serviceName} not found in {cluster}.");
// Print new service events oldest-first (ECS returns newest first).
foreach (var e in (service.Events ?? []).AsEnumerable().Reverse())
{
if (e.Id is not null && seenEvents.Add(e.Id))
{
Console.WriteLine($"{e.CreatedAt:HH:mm:ss} {e.Message}");
}
}
var primary = (service.Deployments ?? []).FirstOrDefault(d => d.Status == "PRIMARY");
if (primary is null)
{
Console.Error.WriteLine("No PRIMARY deployment found.");
return 1;
}
Console.WriteLine(
$"[{DateTime.UtcNow:HH:mm:ss}] {primary.TaskDefinition?.Split('/').Last()} " +
$"rollout={primary.RolloutState} running={primary.RunningCount ?? 0}/{primary.DesiredCount ?? 0} " +
$"failed={primary.FailedTasks ?? 0}");
if (primary.RolloutState == DeploymentRolloutState.COMPLETED)
{
Console.WriteLine("Deployment completed.");
return 0;
}
if (primary.RolloutState == DeploymentRolloutState.FAILED)
{
Console.Error.WriteLine($"Deployment FAILED: {primary.RolloutStateReason}");
return 1;
}
await Task.Delay(TimeSpan.FromSeconds(15), cts.Token);
}
}
catch (OperationCanceledException)
{
Console.Error.WriteLine($"Timed out after {timeout.TotalMinutes} minutes waiting for the rollout.");
return 1;
}Run it after an update:
aws ecs update-service --cluster hl-m03-cluster --service hl-m03-accounts-api --task-definition hl-m03-accounts-api
AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1 dotnet run -- hl-m03-cluster hl-m03-accounts-api 15Code explained line by line
Task definition
"family"— the name; eachregister-task-definitioncall creates the next revision (hl-m03-accounts-api:1,:2…). Revisions are immutable, which gives you an audit trail of exactly what ran."requiresCompatibilities": ["FARGATE"]— ECS validates the definition against Fargate rules at registration time, so mistakes fail early."networkMode": "awsvpc"— required on Fargate. Each task gets its own ENI, private IP and security groups. That is why the target group type isip."cpu": "256","memory": "512"— 0.25 vCPU and 512 MiB, the smallest Fargate size. Fine for a small API; measure before going bigger."runtimePlatform"withARM64— Graviton. Your image must be built forlinux/arm64(the module's Dockerfile cross-compiles). An x64-only image fails withexec format error."executionRoleArn"/"taskRoleArn"— the two roles discussed above. Registering a task definition with a role requiresiam:PassRoleon that role."volumes": [{ "name": "tmp" }]plus"mountPoints"— an ephemeral bind mount so/tmpstays writable while the rest of the file system is read-only. .NET writes some temporary files (and diagnostics) there."readonlyRootFilesystem": true— an attacker who gets code execution cannot drop files into the application folder. This is a common control in CIS-style container benchmarks."portMappings"— container port 8080, the defaultASPNETCORE_HTTP_PORTSin the official .NET 8+ images. Withawsvpcthere is no host port to choose."environment"— non-secret settings only. Double underscores map to configuration sections:Repository__TableNamebecomesRepository:TableName."stopTimeout": 30— after ECS sendsSIGTERM, it waits up to 30 seconds (the maximum on Fargate is 120) beforeSIGKILL. The .NET host's default shutdown timeout is also 30 seconds, so in-flight requests can finish."logConfiguration"with"mode": "non-blocking"— if CloudWatch Logs is slow, log lines are buffered (up tomax-buffer-size) rather than blocking your application'sstdout. The trade-off is that you may lose lines under extreme pressure; for an audit trail, write audit events to a durable store, not only to logs.
Watcher
new AmazonECSClient()— resolves Region and credentials from the default chain; no keys in code.DescribeServicesAsync— one call returns the service, its deployments and its last 100 events.response.Services?.FirstOrDefault()— in AWS SDK for .NET v4, list properties arenull(not empty) when absent and numeric properties are nullable (int?). Code written for v3 often throwsNullReferenceExceptionafter upgrading.service.Events ... .Reverse()— ECS events are newest-first; reversing gives a readable timeline.primary.RolloutState—IN_PROGRESS,COMPLETEDorFAILED. With the circuit breaker on,FAILEDmeans ECS gave up and (with rollback) started redeploying the last good revision.- Exit codes
0/1/2— CI systems understand these without parsing output.
Security considerations
⚠ Learning shortcut — not for production
The lab ALB listens on HTTP port 80 with no WAF, and a single NAT gateway gives the private tasks outbound internet access. In production: an HTTPS 443 listener with an ACM certificate and a modern TLS policy, an HTTP→HTTPS redirect, an AWS WAF web ACL (see WAF and Shield), interface VPC endpoints for ECR (ecr.api, ecr.dkr), CloudWatch Logs, Systems Manager and Secrets Manager plus the S3 gateway endpoint (ECR image layers live in S3), and either no NAT gateway at all or an egress firewall (see segmentation and egress control).
- Least privilege on both roles. Execution role: managed policy plus, if you inject secrets,
secretsmanager:GetSecretValueon those specific secret ARNs. Task role: specific actions on specific resources. - Confused-deputy protection. Add
aws:SourceAccount(and optionallyaws:SourceArn) conditions to the trust policy. - No public IPs on tasks. Enforce it with an SCP or AWS Config rule (
ecs-task-definition-…andecs-service-…managed rules exist ⚠ VERIFY exact names) so a hurried engineer cannot turn it on. - Immutable image tags and digest pinning. With ECR tag immutability,
v1can never be overwritten. For the strongest guarantee, reference the image by digest (@sha256:…). - Non-root, read-only, no shell. The chiseled .NET image runs as UID 1654 with no shell. ECS Exec is therefore of little use; that is a feature, not a bug.
- ECS Exec governance. If you enable
enableExecuteCommandon a service, every session is an interactive shell into production. Log sessions to CloudWatch or S3 with KMS encryption, and restrictecs:ExecuteCommandto a break-glass role. - Logs contain personal data. Even with pseudonymised customer IDs, logs are personal data under UK GDPR. Set retention, encrypt the log group with a KMS key in production, and restrict
logs:GetLogEvents. - Shared responsibility. AWS patches the Fargate host and micro-VM; you patch the image. Rebuild and redeploy at least monthly, and on every critical CVE.
Banking use case
Harbourline runs the Accounts API on Fargate in two AZs in eu-west-2 (London) for UK customers. The platform team publishes a CDK construct library that every team uses, so all services get the same baseline: private subnets, alb-sg → api-sg referencing, VPC endpoints instead of NAT, read-only root file systems, KMS-encrypted log groups with 90-day retention and a circuit breaker with rollback.
A release pipeline registers a new task definition revision, updates the service and runs the rollout watcher. If the rollout fails, the circuit breaker rolls back automatically, the pipeline goes red, and the change ticket is automatically updated with the failure reason from rolloutStateReason. The change advisory board gets evidence of "release attempted, auto-rolled back, no customer impact" without anyone writing it by hand.
Each task definition revision is retained, so in a regulatory inquiry the bank can answer "which image digest, which environment variables and which IAM role served customer requests at 14:32 on 3 March?" by matching the service's deployment history to the revision.
Common mistakes
- Putting application permissions on the execution role → "it worked when I added DynamoDB to that role" → the code does not use the execution role at all, so the change either did nothing or (worse) you also merged the roles. Keep them separate and name them
…-execand…-task. assignPublicIp = ENABLEDto fix image pulls → tasks in private subnets cannot reach ECR without NAT or endpoints → add a NAT gateway or the ECR, S3 and Logs endpoints instead.- Target type
instance→ registration fails forawsvpctasks → useip. - Health check on
/→ the API returns 404 or requires authentication → the ALB marks every task unhealthy. Use an anonymous/health/ready. - Using
latestas the image tag → a deployment of "the same" task definition pulls a different image → use immutable version tags or digests. - Forgetting the deregistration delay → each rolling deployment waits 300 seconds per batch → set 30–60 seconds for short API requests.
- x64 image on an ARM64 task →
exec format errorin the logs → build with--platform linux/arm64or a multi-arch manifest. - No log retention → logs kept forever, growing cost and GDPR storage-limitation problems → set retention on every log group.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Task stops with CannotPullContainerError ... i/o timeout | Private subnet with no route to ECR (no NAT, no endpoints) | Add NAT or the ecr.api, ecr.dkr, S3 gateway and logs endpoints |
CannotPullContainerError ... not found | Wrong tag, wrong Region or the image was never pushed | aws ecr describe-images --repository-name … --image-ids imageTag=v1 |
ResourceInitializationError: unable to pull secrets or registry auth | Execution role lacks secretsmanager:GetSecretValue / ssm:GetParameters / kms:Decrypt, or no network path to those services | Add the permission for that ARN; add endpoints or NAT |
exec format error in logs | Image architecture does not match runtimePlatform | Build for linux/arm64 or switch the task to X86_64 |
| Tasks start, then stop with "failed ELB health checks" | Wrong health check path or port, app not listening on 8080, security group blocks ALB → task | Check the target group's health check, ASPNETCORE_HTTP_PORTS, and api-sg inbound rule from alb-sg |
| Tasks keep restarting during slow start-up | Health check grace period too short | Raise healthCheckGracePeriodSeconds; make start-up faster |
| ALB returns 503 | No healthy targets registered | aws elbv2 describe-target-health --target-group-arn … |
App logs AccessDeniedException for DynamoDB | Task role missing the action or resource ARN | Fix the task role policy; check the ARN includes /index/* for GSIs |
| No logs at all | Execution role cannot write, wrong log group name or Region | Check logConfiguration and the execution role |
services-stable waiter times out | Deployment still cycling | aws ecs describe-services → events; then look at stopped tasks' stoppedReason |
Cost and cleanup
What this lesson costs (us-east-1, checked 2026-11-18; prices change, confirm on the pricing pages):
| Resource | Rate | One hour |
|---|---|---|
| 2 Fargate tasks, ARM64, 0.25 vCPU + 0.5 GB | about USD 0.0099 per task-hour | about USD 0.02 |
| Application Load Balancer | about USD 0.0225 per hour + LCU usage ⚠ VERIFY | about USD 0.03 |
| NAT gateway | about USD 0.045 per hour + USD 0.045 per GB processed | about USD 0.05 |
| Public IPv4 addresses (ALB nodes, NAT) | USD 0.005 per IP per hour | about USD 0.02 |
| CloudWatch Logs | USD 0.50 per GB ingested (first 5 GB/month free tier) ⚠ VERIFY | cents |
Total: roughly USD 0.10–0.15 for one hour. Leaving it running for a month costs about USD 70–100, mostly ALB and NAT. Tear down the same day.
Free alternative: run the same image locally with docker compose (previous lessons) and do this lesson's CLI and IaC sections as a design-only exercise, using cdk synth or terraform plan without deploying.
Teardown:
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# Scale to zero first so the service can be deleted cleanly.
aws ecs update-service --cluster hl-m03-cluster --service hl-m03-accounts-api --desired-count 0
aws ecs delete-service --cluster hl-m03-cluster --service hl-m03-accounts-api --force
aws ecs wait services-inactive --cluster hl-m03-cluster --services hl-m03-accounts-api
aws elbv2 delete-load-balancer --load-balancer-arn "$ALB_ARN"
aws elbv2 wait load-balancers-deleted --load-balancer-arns "$ALB_ARN"
aws elbv2 delete-target-group --target-group-arn "$TG_ARN"
aws ecs delete-cluster --cluster hl-m03-cluster
# Deregister every revision of the task definition (they cost nothing, but keep the account tidy).
for td in $(aws ecs list-task-definitions --family-prefix hl-m03-accounts-api --query 'taskDefinitionArns' --output text); do
aws ecs deregister-task-definition --task-definition "$td" > /dev/null
done
aws logs delete-log-group --log-group-name /hl-m03/accounts-api/ecs
aws iam detach-role-policy --role-name hl-m03-accounts-api-exec \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam delete-role --role-name hl-m03-accounts-api-exec
aws iam delete-role-policy --role-name hl-m03-accounts-api-task --policy-name accounts-api-runtime
aws iam delete-role --role-name hl-m03-accounts-api-task
# If you used the CDK stack instead: cdk destroy HlM03-Fargate
# Delete the NAT gateway and release its Elastic IP if you created them by hand.
# Final check: nothing tagged for this lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query 'ResourceTagMappingList[].ResourceARN'Exercises
- 🟢 Guided: Deploy the Accounts API image with the CLI steps above, call
/health/readyand one account endpoint through the ALB, then register revision 2 withFeatures__IncludePendingTransactions=trueand roll it out while the C# watcher runs. - 🟡 Challenge: Deliberately break revision 3 (set the container port to 9090 in the port mapping only, so the app still listens on 8080). Observe the circuit breaker roll back. Record how long the rollback took and which events ECS emitted.
- 🔴 Live problem: Ticket INC-4471: "After moving the Accounts API to new private subnets, tasks fail to start with
ResourceInitializationError. The old subnets worked. No code changed." Find the cause and propose two fixes with their costs.
Assignment
Create a deploy/ folder in your portfolio repository with: the task definition JSON (role ARNs replaced by placeholders), a Terraform or CDK stack that deploys it, the rollout watcher, and a short README.md that explains the two roles and the network path for image pulls. Include a screenshot-free "evidence" section: the CLI output of describe-services showing rolloutState: COMPLETED.
Solutions
Register the second revision by changing only the environment block, then run aws ecs update-service ... --task-definition hl-m03-accounts-api (no revision number means "latest ACTIVE revision"). The watcher shows a new PRIMARY deployment, two new tasks starting (maximumPercent 200 allows four tasks briefly), the targets passing health checks, the old tasks draining for the 30-second deregistration delay, and finally rollout=COMPLETED. Why it works without downtime: minimumHealthyPercent 100 means ECS never stops an old task until a new one is healthy behind the ALB.
The new tasks start, but the service registers them on port 9090 where nothing listens, so ALB health checks time out. After the grace period, ECS stops them and launches replacements. Once the failed-task count reaches the circuit breaker threshold (at least 3 failures; the exact threshold scales with desired count ⚠ VERIFY the formula), the deployment becomes FAILED and ECS starts a rollback to revision 2. Expect several minutes: grace period + health check intervals × unhealthy threshold, repeated per failed task. Lesson: shorter health checks and grace periods make failure detection faster, but too short causes false failures during start-up. See the health checks lesson.
ResourceInitializationError happens before your code runs, while Fargate pulls the image, secrets or registry credentials using the execution role. "No code changed, new subnets" points at networking: the new private subnets' route table has no route to a NAT gateway, or the VPC endpoints were not associated with the new subnets/security groups (interface endpoints need an ENI in each subnet's AZ and a security group allowing 443 from the tasks). Fix 1: add a route 0.0.0.0/0 → nat-… to the new route table (about USD 33/month per NAT plus data). Fix 2: add interface endpoints for ecr.api, ecr.dkr, logs, secretsmanager/ssm and the S3 gateway endpoint to the new subnets (about USD 7–8/month per endpoint per AZ, but no internet egress — preferred for a bank). Confirm with stoppedReason on a stopped task.
Knowledge check
Answer: the task role. Image pulls and logs prove the execution role works. Your SDK calls use the task role, which lacks the DynamoDB permission or the right resource ARN.
Answer: Fargate requires awsvpc networking, where each task has its own ENI and private IP rather than sharing an EC2 host's address. The ALB therefore routes to IP addresses. instance targets are for EC2 instances; lambda targets are for functions.
Answer: During a deployment ECS may run up to 4 tasks (200%) and must keep at least 2 healthy (100%). It starts new tasks first and stops old ones only after replacements are healthy, so capacity never drops. The cost is briefly paying for double the tasks.
Answer: Only when a new task starts. Injected values are read at container start. Force a new deployment, or have the application fetch the secret itself through a cache (see the secrets lesson).
Answer: (1) A NAT gateway in a public subnet with a default route from the private route table. (2) Interface VPC endpoints for ecr.api and ecr.dkr plus the S3 gateway endpoint (layers are stored in S3) and a CloudWatch Logs endpoint for logging.
Interview questions
- "Explain task role vs execution role." — The execution role is used by the ECS agent to set the task up: pull the image, write logs, fetch secrets referenced in the task definition. The task role is assumed by the application code through the container credentials endpoint. Separate them so a compromised application cannot read every injected secret or pull other images, and so permissions map to responsibilities.
- "Why Fargate over ECS on EC2 for a bank API?" — No hosts to patch, harden or evidence; per-task isolation in a micro-VM; per-second billing; simpler scaling. Choose EC2 when you need GPUs, very high steady utilisation where reserved EC2 is cheaper, special kernel settings or daemon-style agents per host.
- "How do you make deployments safe on ECS?" — Immutable images, rolling updates with
minimumHealthyPercent100, a readiness-based ALB health check, a sensible grace period, the deployment circuit breaker with rollback, CloudWatch alarms on 5xx and latency, and a pipeline step that waits forrolloutState. For high-risk changes, blue/green with a test listener. - "A deployment has been in progress for an hour. What do you check?" — Service events, stopped tasks'
stoppedReasonand exit codes, target health reason codes, application logs for start-up exceptions, and whether the circuit breaker is enabled.
Certification practice
A developer deploys a .NET API on ECS with Fargate. The API must read items from one DynamoDB table. Images are in a private ECR repository. Which configuration follows least privilege?
A. Attach AmazonDynamoDBReadOnlyAccess and AmazonECSTaskExecutionRolePolicy to a single role used as both task role and execution role.
B. Attach AmazonECSTaskExecutionRolePolicy to the execution role; create a task role allowing dynamodb:GetItem and dynamodb:Query on the table ARN.
C. Store an IAM user's access keys as environment variables in the task definition.
D. Attach a DynamoDB policy to the execution role and leave the task role empty.
Answer: B. The agent's needs and the code's needs are separated and scoped. A merges roles and grants read on every table. C uses long-lived keys in plain text. D gives the permission to a role the code never uses, so the calls fail.
A team deploys new task definition revisions with rolling updates. Last week a bad image caused tasks to fail health checks for two hours before someone noticed. What is the simplest way to stop and revert such deployments automatically?
A. Enable the ECS deployment circuit breaker with rollback.
B. Set minimumHealthyPercent to 0.
C. Increase the health check grace period to 3,600 seconds.
D. Use the latest tag so the previous image is always available.
Answer: A. The circuit breaker detects repeated task failures and rolls back to the last completed deployment. B allows ECS to stop all healthy tasks. C delays failure detection by an hour. D makes deployments non-reproducible and does nothing about rollback.
A company runs Fargate tasks in private subnets. Security policy forbids internet egress from these subnets. Tasks must pull images from Amazon ECR and send logs to CloudWatch Logs. Which combination is required? (Choose TWO.)
A. Interface VPC endpoints for ecr.api, ecr.dkr and CloudWatch Logs.
B. A gateway VPC endpoint for Amazon S3.
C. A NAT gateway in each private subnet.
D. assignPublicIp set to ENABLED.
E. An internet gateway route in the private route table.
Answer: A and B. ECR's API and Docker endpoints plus Logs need interface endpoints, and image layers come from S3 through the gateway endpoint. C, D and E all create internet egress, which the policy forbids (and a NAT gateway belongs in a public subnet anyway).
A steady internal API on Fargate uses 10% CPU and 30% memory on 1 vCPU / 2 GB x86 tasks. The image is multi-architecture. What reduces cost the most with the least effort?
A. Move to ECS on EC2 with On-Demand instances. B. Resize tasks to 0.5 vCPU / 1 GB and switch the runtime platform to ARM64. C. Increase the desired count and reduce the task size to 0.25 vCPU / 0.5 GB without testing. D. Move to Lambda with provisioned concurrency.
Answer: B. Right-sizing based on measured usage plus Graviton pricing (about 20% cheaper) cuts cost with a task definition change. A adds host management. C risks out-of-memory kills. D changes the hosting model and provisioned concurrency is billed continuously.
Completion checklist
- I can explain cluster, task definition, task and service to a colleague in two sentences
- I can say which role pulls the image and which role my code uses
- I deployed the API to private subnets behind an ALB (or completed the design-only alternative)
- I watched a rollout complete and a broken rollout roll back
- I ran the teardown and the tag check returned an empty list
Key takeaways
- ECS is the orchestrator; Fargate removes the servers. You still own the image, roles, network and alarms.
- Execution role = the agent's permissions (pull, logs, injected secrets). Task role = your code's permissions. Never merge them.
- Tasks go in private subnets without public IPs, behind an ALB with an
iptarget group; reach AWS services through NAT or (better) VPC endpoints. - Safe deployments:
minimumHealthyPercent100, readiness health checks, a sensible grace period, a short deregistration delay and the circuit breaker with rollback. - An idle ALB and NAT gateway cost more than the tasks. Deploy, test and destroy within the hour.
Further reading
- Amazon ECS task definition parameters for Fargate
- Amazon ECS task execution IAM role
- Amazon ECS task IAM role
- Pass Secrets Manager secrets through Amazon ECS environment variables
- Amazon ECS Express Mode
- CreateService API reference
- Health checks for Application Load Balancer target groups
- AWS Fargate pricing
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Amazon ECR: storing .NET images safely
Use Amazon Elastic Container Registry the way a bank should: private repositories, immutable tags, scan on push and enhanced scanning, lifecycle and archive policies, KMS encryption, cross-account access, private networking, and a C# release gate that blocks images with critical vulnerabilities.
Elastic Beanstalk for .NET
Deploy an ASP.NET Core API to Elastic Beanstalk on the .NET 10 Amazon Linux 2023 platform, understand every resource Beanstalk creates for you, harden the environment, and know when the Windows/IIS platform is the right choice.