Playbook: ECS task keeps restarting
Diagnose an ECS service whose tasks start and stop in a loop: read the stopped reason and exit code, work out whether the task never started, crashed, ran out of memory or was killed by a health check, and fix it safely.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.ECS) · AWS CLI v2 · Amazon ECS on AWS Fargate · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Read an ECS service's events and a stopped task's
stoppedReason,stopCodeand containerexitCodeto decide which of six failure classes you have. - Use a task's lifetime (start to stop) as a clue to separate crashes, health-check kills and slow startups.
- Diagnose the .NET-specific causes: wrong port, missing configuration, unreachable dependencies, no
curlin the image and memory limits. - Fix pull, secret and logging failures at the task execution role and network layers.
- Stabilise a failing deployment with rollback or the deployment circuit breaker, without losing evidence.
- Classify stopped tasks programmatically with the ECS SDK, and configure a service so the next failure is shorter and clearer.
Prerequisites
- Deploy to Fargate and Health checks that help instead of hurt
- Containerising .NET and Secrets management
- ECS deployment strategies and Rollback strategies
- Troubleshooting production issues methodically
- Tools: AWS CLI v2 with the
learn-sandboxprofile; .NET 10 SDK and Docker for the local reproduction
Why this matters in banking
On release night Harbourline's Accounts API gets a new version. The deployment shows "In progress" for an hour. The service says desired count 3, running 0. Tasks start and stop every minute or two. Customers are still being served by the old version only if the rolling deployment kept it up; if the team used a "replace everything" approach, the API is down. Meanwhile each restart pulls the image again, writes a few log lines and vanishes, and the change window is ticking.
A restart loop is dangerous for two reasons. It hides the actual error in a flood of identical events. And the default reaction (increase memory, increase the grace period, restart again) wastes the window. The stopped-task record already tells you which class of failure you have, and ECS keeps stopped tasks visible only for a limited time (roughly an hour, ⚠ VERIFY), so you must capture the evidence first.
The idea in plain language
An ECS service says "keep N copies of this task running". A task goes through a lifecycle: it is provisioned (network interface, image pull, secrets), its containers start, the app becomes healthy, and later it is stopped. A restart loop means one stage fails every time, and the service scheduler starts a replacement, which fails at the same stage.
So the question is always: at which stage does it die?
| Stage | What happens | If it fails |
|---|---|---|
| 1. Provision | ENI created, image pulled, secrets and log streams set up by the execution role | Task never starts: CannotPullContainerError, ResourceInitializationError |
| 2. Start | Container process launches | Immediate exit: bad command, missing file, crash during startup |
| 3. Run | App initialises, binds the port, loads configuration | Exits after a few seconds: unhandled exception, port conflict, missing config |
| 4. Health | Container health check and load balancer health check run | Killed by ECS: Task failed ELB health checks, container marked UNHEALTHY |
| 5. Steady | Serves traffic | Killed for memory (exit 137), or crashes under load |
Analogy: A new hire's first day. Stage 1 is "badge, laptop, accounts": if IT did not provision them, the person never gets in. Stage 2 and 3 are "the person walks in and immediately walks out" (crash). Stage 4 is the manager checking "are you ready to take calls?" every 30 seconds and sending the person home if three checks fail. Stage 5 is the person collapsing at their desk from overwork (out of memory). The time they spent in the building tells you which.
Decode the lifetime
Subtract startedAt from stoppedAt for several stopped tasks. A very short, constant lifetime (a few seconds) is a crash. A lifetime close to healthCheckGracePeriodSeconds plus the unhealthy time is the load balancer killing the task. A longer, variable lifetime that grows with traffic is a memory limit.
What problem the AWS service solves
ECS keeps the desired number of tasks running, replaces failed ones and (with the deployment circuit breaker) can detect a deployment that cannot reach a steady state and roll back. What it does not do is fix your configuration, and its default retry behaviour is to keep trying, which creates a loop.
Practical limits:
- ECS keeps stopped tasks visible only for a limited time. Use
list-tasks --desired-status STOPPEDearly, and send container logs to CloudWatch Logs so evidence survives. - The service event list holds the most recent events only (about 100). Capture it early.
- A Fargate task takes time to provision. A "restart" is not instantaneous: image pull plus ENI plus start can be tens of seconds.
- The default
stopTimeoutis 30 seconds (up to 120 seconds on Fargate, ⚠ VERIFY). A .NET app that does not shut down in time is killed with SIGKILL (exit code 137). - Memory is a hard limit per task and per container: exceeding it kills the container.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Container Apps / App Service "CrashLoopBackOff" in logs | ECS service events and stopped-task reasons | ECS does not back off visibly, so the loop is fast and noisy. Read stoppedReason |
| App Service health check path (removes instance) | ALB target group health check plus healthCheckGracePeriodSeconds | Failing the ALB check makes ECS stop and replace the task. The grace period protects slow starters |
| Managed identity pulls from ACR | Task execution role pulls from ECR and reads secrets | Two roles: the execution role (platform) and the task role (your app). Mixing them up is the top cause of provisioning failures |
| Azure Key Vault reference in app settings | Task definition secrets with valueFrom (Secrets Manager or Parameter Store) | Resolved at task start by the execution role. A failure stops the task before your code runs |
| Log Analytics container logs | awslogs driver to CloudWatch Logs | The log group must exist (or the driver must create it), or the task fails at start |
| Kubernetes liveness and readiness probes | Container health check (liveness-like) and ALB health check (readiness-like) | A failing container health check restarts the container; a failing ALB check replaces the task. See Health checks |
| Deployment slots swap-back | ECS deployment circuit breaker with rollback | Rolls back to the last steady task definition when new tasks cannot start |
Architecture
Where a task can die
Diagnosis flow
Step-by-step implementation
Minutes 0 to 5: stabilise and preserve evidence
- Do not keep clicking Update service. Each force-new-deployment creates a new wave of failing tasks and pushes older evidence out.
- Capture the service events and the stopped tasks now (CLI below) into a file.
- If the service is in a rolling deployment and the old tasks are still serving, leave them; do not scale to zero. If it is fully down and a previous task definition revision was healthy, roll back to it (
update-service --task-definition family:previous-revision), then investigate calmly. If you enabled the deployment circuit breaker with rollback, ECS may do this for you. - Tell stakeholders the state (down or degraded) and the next update time.
Minutes 5 to 15: read the stopped reason and exit code
Use the table below. The stoppedReason text is what ECS recorded; the container reason and exitCode add detail.
| What you see | Meaning | Go to |
|---|---|---|
CannotPullContainerError | Image pull failed: wrong tag, missing in ECR, no route to ECR, or the execution role lacks ECR access | Step on provisioning failures |
ResourceInitializationError: unable to pull secrets or registry auth | Execution role or network cannot reach Secrets Manager, SSM or ECR | Step on provisioning failures |
ResourceInitializationError: failed to validate logger args ... log group does not exist | awslogs log group missing | Create the group, or let the driver create it (needs permission) |
Essential container in task exited with exit code 1, 134, 139 | The process crashed | Read the first log lines |
Task failed ELB health checks in (target-group ...) | The load balancer marked the task unhealthy | Step on health checks |
Container health check UNHEALTHY | The container's own check failed | Step on health checks (and the curl trap) |
Exit code 137, or OutOfMemoryError: Container killed due to memory usage | SIGKILL: memory over the limit, or the app did not stop within stopTimeout | Step on memory |
| Exit code 143 | SIGTERM: ECS stopped it deliberately | Look at why: deployment, scale in, failed health check |
| Exit code 0 | The process ended normally | Wrong entrypoint, or a batch image used as a service |
Capacity is unavailable at this time in events | Fargate has no capacity in that AZ right now (transient) | Retry later, spread across subnets and AZs |
Minutes 15 to 30: fix provisioning failures (task never starts)
If startedAt is missing, your code never ran. Work down the checklist:
- Image: does the tag exist in ECR in this account and Region (
aws ecr describe-images)? Wrong tag after a pipeline change is the most common cause. - Execution role: has
AmazonECSTaskExecutionRolePolicyor equivalent (ECR pull, log stream creation), plussecretsmanager:GetSecretValue(orssm:GetParameters) on the specific secret ARNs the task definition uses, pluskms:Decrypton the key if the secret uses a customer managed key. - Network path to ECR, Secrets Manager and CloudWatch Logs: a Fargate task in a private subnet needs a NAT gateway or VPC endpoints:
ecr.api,ecr.dkr, an S3 gateway endpoint (image layers),logsandsecretsmanageras needed. A missing endpoint looks like a timeout in the stopped reason. - Secret reference format:
valueFrommust be the secret ARN, optionally with a JSON key suffix (arn:...:secret:name-AbCdEf:jsonKey::). A typo or the wrong Region fails the task. - Security group: outbound 443 allowed (default allows all, hardened groups may not).
Minutes 30 to 45: fix crashes and health-check kills
Crash on startup. Open the first lines of the task's log stream (the log stream name contains the task ID). Typical .NET causes: OptionsValidationException from ValidateOnStart (missing configuration), a connection string that is empty because a secret key name is wrong, unreachable database (timeout at start), file path differences on Linux (case-sensitive), or a wrong ENTRYPOINT.
Health-check kill. Check, in order:
- Port: .NET 8 and later container images listen on 8080 by default (not 80). The task definition
containerPort, the target group port andASPNETCORE_HTTP_PORTSmust agree. - Path and status: the target group health check path (for example
/health/ready) must return a status in the success range (200 by default). A redirect to HTTPS (301) or a401fails it. - Grace period:
healthCheckGracePeriodSecondson the service must cover startup time (image pull is not counted but app init is). A .NET app with EF Core migrations or a cold JIT might need 60 to 90 seconds. - Dependency in the readiness endpoint: if
/healthchecks the database and the database is slow, every task is killed. Use a cheap liveness path for the load balancer and a separate readiness endpoint. See Health checks. - The curl trap: a container health check of
CMD-SHELL curl -f http://localhost:8080/healthfails if the image has nocurlor no shell. Chiseled and distroless .NET images have neither, so the check never passes, and ECS kills the task. Use the ALB health check, or a built-in check.
Minutes 45 to 60: memory and shutdown
For exit 137: compare the task's memory limit with the app's real use (CloudWatch MemoryUtilization, or Container Insights). .NET respects the container limit: by default the garbage collector's heap hard limit is a percentage (75 percent) of the container memory limit (⚠ VERIFY), so a 512 MB task leaves about 384 MB of managed heap, and native memory and the runtime come out of the rest. Raising memory is a valid fix, but first look for a leak or an unbounded cache. For a 137 that happens during deployments, the app is not stopping within stopTimeout: handle SIGTERM (the host does this by default), keep HostOptions.ShutdownTimeout shorter than stopTimeout, and finish in-flight work.
After the incident: prevent and record
Turn on the deployment circuit breaker with rollback, set the log retention, add alarms on RunningTaskCount below desired and on the service event for failed deployments, and record: the stopped reason, the exit code, the root cause and the change that fixed it.
AWS Console instructions
- Console → ECS → Clusters → (cluster) → Services → (service) → Events tab. Read the newest messages (for example
has started 1 tasks,has reached a steady state,failed ELB health checks). - Console → ECS → Clusters → (cluster) → Tasks and filter by Desired status: Stopped. Open a task: the Stopped reason, the Containers table (exit code, reason) and the Logs tab.
- Console → ECS → Task definitions → (family) → (revision) → JSON to check port mappings, memory, health check, secrets and the execution role.
- Console → EC2 → Target Groups → (group) → Targets (health status and reason, for example
Request timed outorHealth checks failed with these codes: [404]), and Health checks to see the path, interval and thresholds. - Console → CloudWatch → Log groups → (group) and open the stream for the stopped task ID.
- Console → VPC → Endpoints to confirm the interface endpoints for ECR, logs and Secrets Manager (or a NAT route) exist for private subnets.
AWS CLI
# 1. Service state and the newest events (capture these early; only about 100 are kept).
aws ecs describe-services --cluster hl-accounts --services hl-accounts-api \
--query 'services[0].{Desired:desiredCount,Running:runningCount,Pending:pendingCount,Deployments:deployments[].{Id:id,Status:status,Rollout:rolloutState,Reason:rolloutStateReason,Td:taskDefinition},Events:events[0:10].message}' \
--profile learn-sandbox
# 2. Stopped tasks for the service (visible for a limited time).
aws ecs list-tasks --cluster hl-accounts --service-name hl-accounts-api --desired-status STOPPED \
--query 'taskArns' --output text --profile learn-sandbox
# 3. Why did they stop? Reason, stop code, exit codes, start and stop times.
aws ecs describe-tasks --cluster hl-accounts --tasks <task-arn-1> <task-arn-2> \
--query 'tasks[].{Started:startedAt,Stopped:stoppedAt,StopCode:stopCode,Reason:stoppedReason,Containers:containers[].{Name:name,Exit:exitCode,Reason:reason,Health:healthStatus}}' \
--profile learn-sandbox
# 4. Target group health: which targets are unhealthy and why.
aws elbv2 describe-target-health --target-group-arn <tg-arn> \
--query 'TargetHealthDescriptions[].{Target:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason,Desc:TargetHealth.Description}' \
--profile learn-sandbox
# 5. Does the image tag the task definition uses exist?
aws ecr describe-images --repository-name hl/accounts-api --image-ids imageTag=2026.11.13.1 --profile learn-sandbox
# 6. Memory pressure: max memory utilisation over the last hours.
aws cloudwatch get-metric-statistics --namespace AWS/ECS --metric-name MemoryUtilization \
--dimensions Name=ClusterName,Value=hl-accounts Name=ServiceName,Value=hl-accounts-api \
--start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Maximum --profile learn-sandbox
# 7. Roll back to the previous known-good task definition revision.
aws ecs update-service --cluster hl-accounts --service hl-accounts-api \
--task-definition hl-accounts-api:41 --profile learn-sandbox
# 8. Get a shell in a running task to test from inside (ECS Exec must be enabled).
aws ecs execute-command --cluster hl-accounts --task <task-id> --container app \
--interactive --command "/bin/sh" --profile learn-sandboxA Logs Insights query for the application log group, for the first error after each start:
fields @timestamp, @logStream, @message
| filter @message like /Unhandled exception|OptionsValidationException|Failed to bind|Connection refused|AccessDenied/
| sort @timestamp desc
| limit 30Infrastructure as Code
A service configured so failures are quick and visible: circuit breaker with rollback, a grace period, an ALB health check on a cheap path, secrets by ARN and logs with retention.
using Amazon.CDK;
using Amazon.CDK.AWS.ECR;
using Amazon.CDK.AWS.ECS;
using Amazon.CDK.AWS.ECS.Patterns;
using Amazon.CDK.AWS.ElasticLoadBalancingV2;
using Amazon.CDK.AWS.Logs;
using Constructs;
public class AccountsApiServiceStack : Stack
{
public AccountsApiServiceStack(Construct scope, string id, ICluster cluster, IStackProps props)
: base(scope, id, props)
{
var repo = Repository.FromRepositoryName(this, "Repo", "hl/accounts-api");
var dbSecret = Amazon.CDK.AWS.SecretsManager.Secret.FromSecretNameV2(this, "DbSecret", "hl/accounts/db");
var logGroup = new LogGroup(this, "Logs", new LogGroupProps
{
LogGroupName = "/ecs/hl-accounts-api",
Retention = RetentionDays.ONE_MONTH,
RemovalPolicy = RemovalPolicy.DESTROY // learning only
});
var service = new ApplicationLoadBalancedFargateService(this, "Service", new ApplicationLoadBalancedFargateServiceProps
{
Cluster = cluster,
ServiceName = "hl-accounts-api",
DesiredCount = 2,
Cpu = 512,
MemoryLimitMiB = 1024,
HealthCheckGracePeriod = Duration.Seconds(90), // covers .NET startup
CircuitBreaker = new DeploymentCircuitBreaker { Rollback = true },
TaskImageOptions = new ApplicationLoadBalancedTaskImageOptions
{
Image = ContainerImage.FromEcrRepository(repo, "2026.11.13.1"),
ContainerPort = 8080, // .NET 8+ images listen on 8080
Environment = new Dictionary<string, string> { ["ASPNETCORE_HTTP_PORTS"] = "8080" },
Secrets = new Dictionary<string, Amazon.CDK.AWS.ECS.Secret>
{
["ConnectionStrings__Accounts"] = Amazon.CDK.AWS.ECS.Secret.FromSecretsManager(dbSecret, "connectionString")
},
LogDriver = LogDrivers.AwsLogs(new AwsLogDriverProps
{
StreamPrefix = "accounts-api",
LogGroup = logGroup
})
}
});
// A cheap liveness-style path. Do not check the database here.
service.TargetGroup.ConfigureHealthCheck(new Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck
{
Path = "/health/live",
Interval = Duration.Seconds(15),
Timeout = Duration.Seconds(5),
HealthyThresholdCount = 2,
UnhealthyThresholdCount = 3
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-ecs");
}
}Complete code
Classify stopped tasks with the ECS SDK
namespace Harbourline.Ecs;
public enum FailureClass { Provisioning, ElbHealthCheck, ContainerHealthCheck, OutOfMemoryOrKilled, AppCrash, NormalExit, StoppedByEcs, Capacity, Unknown }
public sealed record Diagnosis(FailureClass Class, string Why, string NextStep);
public static class StoppedTaskClassifier
{
public static Diagnosis Classify(string? stoppedReason, int? exitCode, string? containerReason,
TimeSpan? lifetime, bool everStarted)
{
var reason = stoppedReason ?? "";
var creason = containerReason ?? "";
if (!everStarted || reason.Contains("CannotPullContainerError", StringComparison.OrdinalIgnoreCase)
|| reason.Contains("ResourceInitializationError", StringComparison.OrdinalIgnoreCase))
return new(FailureClass.Provisioning,
"The containers never started, so your code did not run.",
"Check the image tag in ECR, the execution role (ECR, secrets, KMS), the log group and the route or VPC endpoints.");
if (reason.Contains("ELB health checks", StringComparison.OrdinalIgnoreCase))
return new(FailureClass.ElbHealthCheck,
"The load balancer marked the task unhealthy and ECS replaced it.",
"Check container port versus target group port, the health path and status, the grace period and whether the path depends on a slow dependency.");
if (creason.Contains("health", StringComparison.OrdinalIgnoreCase) && reason.Contains("Essential", StringComparison.OrdinalIgnoreCase))
return new(FailureClass.ContainerHealthCheck,
"The container health check failed.",
"Does the image contain the tool the check uses (curl, a shell)? Chiseled and distroless images do not. Use the ALB check or a built-in check.");
if (exitCode == 137 || creason.Contains("OutOfMemory", StringComparison.OrdinalIgnoreCase))
return new(FailureClass.OutOfMemoryOrKilled,
"SIGKILL: out of memory, or the app did not stop within stopTimeout.",
"Compare MemoryUtilization with the task limit. If this happens only during deployments, fix graceful shutdown.");
if (exitCode is 1 or 134 or 139 || (exitCode is not null and not 0 and not 143))
return new(FailureClass.AppCrash,
$"The process exited with code {exitCode}. {(lifetime is { } l ? $"It lived {l.TotalSeconds:F0} s." : "")}",
"Read the first lines of the task's log stream: missing configuration, unreachable dependency, bad port or an unhandled exception.");
if (exitCode == 143)
return new(FailureClass.StoppedByEcs, "ECS sent SIGTERM and the app exited.",
"Find out why ECS stopped it: deployment, scale in, or a failed health check.");
if (exitCode == 0)
return new(FailureClass.NormalExit, "The process ended normally.",
"Is the entrypoint a batch or console app? A service container must keep running.");
if (reason.Contains("Capacity is unavailable", StringComparison.OrdinalIgnoreCase))
return new(FailureClass.Capacity, "Fargate had no capacity in that zone at that moment.",
"Use several subnets across Availability Zones and retry.");
return new(FailureClass.Unknown, "Not enough information.", "Capture the full task JSON and the service events.");
}
}using Amazon.ECS;
using Amazon.ECS.Model;
using Task = System.Threading.Tasks.Task; // Amazon.ECS.Model also defines a Task type
namespace Harbourline.Ecs;
public sealed class StoppedTaskReport(IAmazonECS ecs)
{
public async Task PrintAsync(string cluster, string service, CancellationToken ct)
{
var list = await ecs.ListTasksAsync(new ListTasksRequest
{
Cluster = cluster, ServiceName = service, DesiredStatus = DesiredStatus.STOPPED, MaxResults = 50
}, ct);
if (list.TaskArns is null || list.TaskArns.Count == 0)
{
Console.WriteLine("No stopped tasks are visible (they are kept for a limited time).");
return;
}
var described = await ecs.DescribeTasksAsync(new DescribeTasksRequest
{
Cluster = cluster, Tasks = list.TaskArns
}, ct);
foreach (var t in described.Tasks.OrderByDescending(t => t.StoppedAt))
{
var container = t.Containers?.FirstOrDefault();
TimeSpan? lifetime = t.StartedAt is { } s && t.StoppedAt is { } e ? e - s : null;
var d = StoppedTaskClassifier.Classify(t.StoppedReason, container?.ExitCode, container?.Reason,
lifetime, everStarted: t.StartedAt is not null);
Console.WriteLine($"{t.TaskArn.Split('/').Last()[..8]} {d.Class,-22} exit={container?.ExitCode?.ToString() ?? "-"} " +
$"life={(lifetime is null ? "-" : $"{lifetime.Value.TotalSeconds:F0}s")}");
Console.WriteLine($" reason: {t.StoppedReason}");
Console.WriteLine($" next: {d.NextStep}");
}
var lifetimes = described.Tasks
.Where(t => t.StartedAt is not null && t.StoppedAt is not null)
.Select(t => (t.StoppedAt!.Value - t.StartedAt!.Value).TotalSeconds).ToList();
if (lifetimes.Count > 2 && lifetimes.Max() - lifetimes.Min() < 5)
Console.WriteLine($"\nLifetimes are nearly constant ({lifetimes.Average():F0}s): a deterministic failure at one stage, not load.");
}
}The ASP.NET Core side: cheap liveness, honest readiness, validated config
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;
var builder = WebApplication.CreateBuilder(args);
// Fail fast and loudly at startup if configuration is wrong: the log shows WHY the task exits.
builder.Services.AddOptions<AccountsOptions>()
.Bind(builder.Configuration.GetSection("ConnectionStrings"))
.Validate(o => !string.IsNullOrWhiteSpace(o.Accounts), "ConnectionStrings:Accounts is missing")
.ValidateOnStart();
builder.Services.AddHealthChecks()
.AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
.AddCheck<DatabaseReadinessCheck>("database", tags: ["ready"]);
// Shut down within the ECS stopTimeout (30 s default).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(20));
var app = builder.Build();
// ALB target group points here: no dependencies, answers in milliseconds.
app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = c => c.Tags.Contains("live") });
// Used by deployment gates and humans, not by the load balancer's kill decision.
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = c => c.Tags.Contains("ready") });
app.MapGet("/", () => "Harbourline Accounts API");
app.Run();
public sealed class AccountsOptions { public string Accounts { get; set; } = ""; }
public sealed class DatabaseReadinessCheck : IHealthCheck
{
public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
=> Task.FromResult(HealthCheckResult.Healthy("stub: replace with a real, cheap query"));
}Tests for the classifier
using Harbourline.Ecs;
using Xunit;
public class StoppedTaskClassifierTests
{
[Fact]
public void Missing_start_time_is_a_provisioning_failure() =>
Assert.Equal(FailureClass.Provisioning,
StoppedTaskClassifier.Classify("Task failed to start", null, null, null, everStarted: false).Class);
[Fact]
public void Elb_kill_is_recognised() =>
Assert.Equal(FailureClass.ElbHealthCheck,
StoppedTaskClassifier.Classify("Task failed ELB health checks in (target-group arn:aws:...)", 143, null,
TimeSpan.FromSeconds(150), true).Class);
[Fact]
public void Exit_137_is_memory_or_forced_stop() =>
Assert.Equal(FailureClass.OutOfMemoryOrKilled,
StoppedTaskClassifier.Classify("Essential container in task exited", 137, "OutOfMemoryError: Container killed due to memory usage",
TimeSpan.FromMinutes(20), true).Class);
[Theory]
[InlineData(1)] [InlineData(134)] [InlineData(139)]
public void Crash_codes_are_app_crashes(int code) =>
Assert.Equal(FailureClass.AppCrash,
StoppedTaskClassifier.Classify("Essential container in task exited", code, null, TimeSpan.FromSeconds(4), true).Class);
}Code explained line by line
everStartedcomes fromstartedAt. A task with no start time never ran your code, so it is a provisioning problem whatever the exit code says.- The ELB check is matched before the exit code because a task killed by the load balancer usually shows exit 143 (SIGTERM) or 137, which would otherwise look like an unrelated stop.
- Exit 137 is treated as SIGKILL: memory over the limit or a shutdown that did not finish within
stopTimeout. Exit 143 is SIGTERM, a deliberate stop. - Exit codes 1, 134 and 139 are crash signatures. An unhandled .NET exception commonly ends with 134 (SIGABRT) (⚠ VERIFY for your runtime image), 139 is a segmentation fault, 1 is a failure the app chose.
StoppedTaskReportlists stopped tasks, describes them in one call (up to 100 ARNs per describe) and prints one line per task with the class and next step. The near-constant lifetime hint at the end flags a deterministic failure rather than a load or leak problem.- In
Program.cs,ValidateOnStartturns a missing connection string into a clear startup exception in the log, not a vague timeout later. - Two health endpoints:
/health/livehas no dependencies and is what the ALB checks;/health/readycan test the database and is for humans and deployment gates. This prevents a database blip from making the load balancer kill every task. ShutdownTimeoutof 20 seconds is below the default ECSstopTimeoutof 30 seconds, so the app finishes before ECS escalates to SIGKILL.
Security considerations
⚠ Learning shortcut — not for production
Debugging a failing task tempts people to add AdministratorAccess to the execution role, or to put secrets directly into the task definition's environment block "just to get it started". Neither is acceptable. Give the execution role access to the specific secret ARNs only, keep secrets in secrets (not environment), and enable ECS Exec only temporarily, with logging to CloudWatch or S3, because it opens an interactive session into a production task.
- Two roles, two jobs. The execution role pulls images and resolves secrets for the platform; the task role is what your app code uses. Keep them separate and least privilege.
- Task definitions are readable. Anyone who can describe a task definition sees environment variables in clear text. Never place credentials there.
- ECS Exec gives shell access. Restrict it with an IAM policy (
ecs:ExecuteCommandon named clusters and tasks), audit it in CloudTrail and enable session logging. Turn it off when finished. - Logs can leak. A startup exception may print a connection string. Do not log configuration values; validate and log only the key name that is missing.
- Do not widen the security group or move the task to a public subnet to "make the pull work". Add the VPC endpoints or NAT route instead.
- Rollback is a change. Use the normal emergency change record. Do not skip approval just because the service is down; follow your bank's emergency process.
Banking use case
Harbourline's Accounts API serves the mobile app. On release night the new revision started and stopped every 70 seconds. The stopped tasks showed Task failed ELB health checks, exit code 143 and a lifetime of about 130 seconds, which matched a 60-second grace period plus two 15-second checks. The logs showed the app ready in 12 seconds, yet the target group reported Health checks failed with these codes: [404]. The cause: the pipeline had changed the health route from /health to /health/live, but the target group still pointed to the old path. The team fixed the target group in IaC, redeployed through the pipeline and the circuit breaker confirmed a steady state. The post-incident actions were a contract test that calls the health path after deployment, and a note in the runbook: "lifetime close to grace plus check time means the load balancer is killing it".
Common mistakes
- Raising memory first. Exit 137 may be a shutdown problem, and other exit codes are not memory at all.
- Increasing the grace period to hide a wrong path or port. The task just stays broken longer.
- Mixing up the execution role and the task role. Pull and secret errors need the execution role; app AWS calls need the task role.
- Health check that tests the database. One database blip restarts every task at once. Use a cheap liveness path for the ALB.
- A container health check that needs curl in a chiseled or distroless image.
- Port 80 in the task definition and 8080 in the app. .NET 8+ images listen on 8080 by default.
- Force-new-deployment loops. They destroy evidence and waste the change window.
- Missing log group or retention. Without logs you debug blind; without retention you pay for logs forever.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
CannotPullContainerError ... not found | Image tag does not exist or the repository is in another account or Region | Check ecr describe-images and the tag in the task definition |
CannotPullContainerError ... i/o timeout | No route to ECR from a private subnet | Add NAT or the ecr.api, ecr.dkr and S3 gateway endpoints |
ResourceInitializationError ... unable to pull secrets | Execution role lacks GetSecretValue, KMS decrypt, or no path to Secrets Manager | Grant the execution role the specific secret and key, add the endpoint |
ResourceInitializationError ... log group does not exist | awslogs group missing | Create it, or add awslogs-create-group with permission |
| Tasks live exactly grace plus about 45 to 60 seconds | ALB health check failing | Fix port, path, status code, or security group between ALB and task |
| Tasks die in 2 to 10 seconds, exit 134 or 1 | Crash at startup | Read the first log lines, check required configuration and secrets |
UNHEALTHY container, no curl | Health check command not in the image | Remove the container check or use an in-process check |
Exit 137 after hours, rising MemoryUtilization | Memory leak or unbounded cache | Profile, cap caches, then right-size memory |
| Exit 137 only during deployments | Slow shutdown beyond stopTimeout | Shorten ShutdownTimeout, drain requests, raise stopTimeout up to the Fargate maximum |
| Deployment rolled back automatically | Circuit breaker detected tasks failing to reach steady state | Read the service events and the stopped reasons of the failed deployment |
| Events show repeated placement failures | Fargate capacity or subnet IP exhaustion | Add subnets in more AZs, check available IPs in the subnets |
Cost and cleanup
Cost label: credits (optional hands-on). The free alternative is to run the image locally with Docker, with the same environment variables, and read the failure from the console; docker run --memory 256m reproduces memory kills, and a wrong port reproduces health failures. If you deploy to the sandbox, Fargate bills per vCPU-second and GB-second (a small task for 30 minutes is a few cents, ⚠ VERIFY current prices), and an ALB bills hourly plus LCU (roughly a few cents per hour in us-east-1, ⚠ VERIFY). A NAT gateway, if you create one, also bills hourly. Deploy, break, diagnose, fix and destroy within an hour.
# CDK
cdk destroy AccountsApiServiceStack --profile learn-sandbox
# Or by hand: scale to zero, then delete the service, load balancer and target group
aws ecs update-service --cluster hl-accounts --service hl-accounts-api --desired-count 0 --profile learn-sandbox
aws ecs delete-service --cluster hl-accounts --service hl-accounts-api --force --profile learn-sandbox
aws logs delete-log-group --log-group-name /ecs/hl-accounts-api --profile learn-sandbox
# Confirm nothing tagged for this lesson remains, and check for leftover NAT gateways or load balancers
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-ecs --profile learn-sandbox
aws elbv2 describe-load-balancers --query 'LoadBalancers[].LoadBalancerName' --profile learn-sandboxExercises
- 🟢 Guided: Run
StoppedTaskClassifierTests. Then run a container locally that needs a variable and exits when it is missing (for exampledocker run --rm -e ASPNETCORE_HTTP_PORTS=8080 your-imagewithout the connection string, using theValidateOnStartcode) and read the exception. Identify which class and exit code it would be on ECS. - 🟡 Challenge: Add a rule to
StoppedTaskReportthat compares the lifetimes with the service'shealthCheckGracePeriodSeconds(passed as a parameter) and prints "matches the grace period plus check time: the load balancer is killing the task" when the median lifetime is within 20 percent ofgrace + interval * unhealthyThreshold. - 🔴 Live problem: The service shows desired 3, running 0. Stopped tasks show
Task failed ELB health checks, lifetime about 100 seconds. The app log shows "Now listening on: http://[::]:8080" at second 8. Target health saysRequest timed out. The container port in the task definition is 8080 and the target group port is 80. What is wrong, what do you change, and how do you make sure it cannot happen again?
Assignment
Write a blameless incident report for a simulated ECS restart loop (use the local reproduction or the optional sandbox deployment) and commit it as incident-reports/ecs-restart-loop.md. Include the stopped-task table, the class, the lifetime reasoning, the fix in IaC, and the guardrail you added (circuit breaker, deployment smoke test). Add StoppedTaskClassifier with tests to your portfolio.
Solutions
The container exits immediately with the OptionsValidationException message ConnectionStrings:Accounts is missing. On ECS it would appear as Essential container in task exited with exit code 134 or 1 depending on how the host terminates (⚠ VERIFY on your image), a lifetime of a few seconds, and class AppCrash. Why: ValidateOnStart makes the failure deterministic and visible in the first log lines.
Compute expected = grace + interval * unhealthyThreshold (for example 60 + 15 * 3 = 105 s). Take the median lifetime of tasks with a start and stop time. If abs(median - expected) / expected <= 0.2, print the message. Why: a median close to that figure is the signature of a load-balancer kill; random or very short lifetimes point elsewhere.
The target group sends health checks to port 80 on the task, where nothing listens, so the check times out and ECS stops the task after the grace period. Fix: make the target group port 8080 (or register the container port in the service's load balancer block with 8080) in the IaC, redeploy via the pipeline, and confirm target health turns healthy. Prevent it: generate the port from one variable shared by the task definition, the target group and ASPNETCORE_HTTP_PORTS; add a post-deployment smoke test that calls the health path through the load balancer; keep the circuit breaker with rollback on. Why: the lifetime (about 100 seconds) and "Request timed out" show the checks never reached the app, and the log proves the app was healthy on 8080.
Knowledge check
Answer: At image and network access: the tag exists in ECR in the right account and Region, the execution role can pull, and the subnet has NAT or the ECR and S3 endpoints. The app never ran, so application logs will not help.
Answer: The load balancer health check fails: wrong port or path, a non-200 status, or a security group blocking the ALB. The lifetime matches grace period plus the time to fail enough checks.
Answer: The app does not stop within stopTimeout, so ECS escalates from SIGTERM to SIGKILL. Fix graceful shutdown and keep the host shutdown timeout shorter than stopTimeout.
Answer: Chiseled and distroless images contain neither a shell nor curl, so the check command cannot run and the container is always marked unhealthy. Use the ALB health check or a check built into the app.
Answer: The task execution role pulls images, writes logs and resolves secrets for the platform. The task role is what your application code uses for its own AWS calls.
Interview questions
- An ECS service cannot keep tasks running. How do you start? Read service events, list stopped tasks and read
stoppedReason,stopCode, exit codes and lifetimes, and decide whether the task never started, crashed, failed health checks or was killed for memory. - How do you tell a crash from a health-check kill? Crash: very short, constant lifetime and a non-zero exit code with errors in the first log lines. Health-check kill: lifetime near grace period plus check time, reason mentions ELB health checks, exit 143 or 137.
- What are the two IAM roles of an ECS task? The execution role (image pull, logs, secrets) and the task role (application permissions).
- How would you make deployments safer? Deployment circuit breaker with rollback, a post-deployment smoke test, cheap liveness checks, proper grace periods and immutable image tags.
- Why separate liveness and readiness? Liveness says "restart me if broken"; readiness says "do not send me traffic yet". Mixing them lets a dependency blip trigger restarts of healthy processes.
Certification practice
An ECS service on Fargate behind an Application Load Balancer repeatedly starts new tasks that are stopped after about two minutes with Task failed ELB health checks. The application logs show it started successfully on port 8080. The target group health check is on port 80. Which action resolves the issue?
A. Increase the task memory to 8 GB. B. Change the target group (and service load balancer mapping) to port 8080 so health checks reach the application. C. Disable the load balancer health check. D. Switch the service to the EC2 launch type.
Answer: B. The check targets a port where nothing listens. A is unrelated, C would remove a protective control and send traffic to broken tasks, and D does not address the port mismatch.
A new task definition revision fails with ResourceInitializationError: unable to pull secrets or registry auth. The task runs in a private subnet with no NAT gateway. The execution role has permission to read the secret. What is the most likely missing piece?
A. A VPC interface endpoint for Secrets Manager (and ECR and CloudWatch Logs), or a route through a NAT gateway. B. A larger CPU reservation. C. A public IP on the application load balancer. D. A longer health check interval.
Answer: A. The task cannot reach the services that the execution role needs. B, C and D do not affect provisioning network access.
A team wants a failing ECS deployment to stop and return to the last healthy task definition automatically, without manual intervention. Which feature is designed for this?
A. An ECS deployment circuit breaker with rollback enabled. B. AWS Config conformance packs. C. An S3 lifecycle rule. D. Amazon GuardDuty.
Answer: A. The circuit breaker detects tasks that cannot reach steady state and rolls back. B evaluates configuration compliance, C manages object storage and D detects threats.
Completion checklist
- I can name the five stages where a task can fail and one error for each
- I can read a stopped task: reason, stop code, exit code and lifetime
- I know why a chiseled image breaks a curl health check
- I ran the classifier tests and added one case
- I can state the port, path and grace-period checks for a health-check kill
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-ecsremains
Key takeaways
- A restart loop is one stage failing repeatedly. Find the stage from
startedAt, the stopped reason, the exit code and the lifetime. - Capture service events and stopped tasks early; they expire.
- Provisioning errors belong to the execution role and the network path, not your code.
- Health-check kills are port, path, status or grace-period problems; keep the ALB path dependency-free.
- Turn on the circuit breaker with rollback so the next bad deployment fixes itself and tells you why.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: KMS AccessDeniedException
Diagnose a KMS AccessDeniedException by checking the key policy, IAM, grants, encryption context, key state and cross-account rules in order, and tell service-principal and SSE-KMS failures apart from ordinary IAM gaps.
Playbook: Lambda timeouts and cold starts
Tell a Lambda timeout from a cold start from throttling using REPORT lines and Logs Insights, find the slow phase (init, your code or a downstream call), and fix it with deadline-aware code, right-sized memory and the right start-up option for a .NET function.