Health checks that help instead of hurt
Design liveness and readiness endpoints in ASP.NET Core, wire them correctly to ALB target groups, ECS container health checks and Elastic Beanstalk, tune timings and grace periods, shut down gracefully, and avoid the restart loop that turns a database blip into a full outage.
Assumes: .NET 10 LTS · ASP.NET Core health checks · Microsoft.AspNetCore.Mvc.Testing 10.0.15 · xunit.v3 4.0.3 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain the difference between liveness, readiness and start-up checks, and which AWS component should call each.
- Implement health endpoints in ASP.NET Core with tags, predicates, safe response bodies and cached dependency checks.
- Configure ALB target group health checks, ECS container health checks, the ECS health check grace period and Elastic Beanstalk health settings — and calculate how long detection and recovery take.
- Recognise and prevent the restart-loop trap, where a shared dependency outage makes the orchestrator kill every healthy task.
- Implement graceful shutdown so deployments and scale-in do not drop in-flight requests.
- Test health behaviour with
WebApplicationFactoryand xUnit, without AWS.
Prerequisites
- Deploying .NET to ECS on Fargate and/or Elastic Beanstalk for .NET.
- Configuration done right —
ValidateOnStartand options. - Tools: .NET 10 SDK, Docker (optional), AWS CLI v2 with
learn-sandbox(only for the optional deployment).
Why this matters in banking
At 09:02 on a Monday, Harbourline's PostgreSQL primary fails over to its standby. The failover takes 40 seconds — well within the database's design. But the Accounts API's health endpoint runs SELECT 1 against the database, and the load balancer is configured to check it every 10 seconds with an unhealthy threshold of 2. Within 20 seconds every task is marked unhealthy. ECS, doing exactly what it was told, stops every task and starts new ones. The new tasks take 25 seconds to start, then also fail their health checks because the database is still failing over. By the time the database is back, ECS is in a replacement loop, the deployment circuit breaker has tripped on an unrelated release, and the 40-second database event has become an 11-minute customer-facing outage on the mobile app.
The reverse also happens: an endpoint that always returns 200 keeps sending traffic to a task whose thread pool is exhausted, so 25% of requests time out for an hour.
Health checks are small pieces of code with outsized influence. Operational-resilience reviews (and regulators' expectations that banks stay within impact tolerances for important business services) make "how does the platform decide an instance is broken?" a real question with a documented answer.
The idea in plain language
A health check is a question the platform asks your application on a schedule. The trick is that there are different questions, asked by different components, with different consequences:
| Check | Question | Who asks | Consequence of "no" |
|---|---|---|---|
| Liveness | "Is this process alive and not stuck?" | Container orchestrator (ECS container health check) | The container/task is killed and replaced |
| Readiness | "Should this instance receive traffic right now?" | Load balancer (ALB target group) | Traffic is routed elsewhere — but on ECS, a task the ALB marks unhealthy is also replaced |
| Start-up | "Have you finished starting?" | Grace periods (ECS healthCheckGracePeriodSeconds, container startPeriod) | Failures are ignored until start-up ends |
| Dependency / deep check | "Are my downstream systems OK?" | Monitoring, dashboards, alarms | A human or an alarm decides — not automatic replacement |
The golden rule: only fail a check whose consequence would actually fix the problem. Restarting a task fixes a deadlocked process. Restarting a task does not fix a database outage — every replacement connects to the same database.
Analogy: A bank branch has a receptionist (load balancer) and a regional manager (orchestrator). The receptionist asks each cashier "can you take a customer?" — if a cashier is on a break, customers go to someone else (readiness). The regional manager asks "are you actually at work?" — if a cashier is missing, the manager sends a replacement (liveness). If the card-payment network is down, the receptionist should still send customers to cashiers who can help with other things, and the manager should certainly not fire and replace every cashier: the network is not their fault.
What problem AWS health checks solve — and how each one behaves
Application Load Balancer target group health checks (checked 2026-11-18):
| Setting | Range | Default (ip/instance targets) |
|---|---|---|
HealthCheckIntervalSeconds | 5–300 | 30 |
HealthCheckTimeoutSeconds | 2–120 | 5 |
HealthyThresholdCount | 2–10 | 5 |
UnhealthyThresholdCount | 2–10 | 2 |
Matcher (success codes) | 200–499 | 200 |
HealthCheckPath | any | / |
- A new target must pass health checks before receiving traffic.
- Fail-open: if all targets in a target group are unhealthy, the ALB routes to all of them anyway. This protects you from a bad check taking everything offline at the routing layer — but not from ECS replacing the tasks.
- Each ALB node checks each target independently, so with several AZs a task receives several health requests per interval. Keep the endpoint cheap.
- Deregistration delay (default 300 seconds) is how long the ALB keeps in-flight connections to a deregistering target.
ECS service behaviour with a load balancer: a task is healthy only if it is RUNNING and the load balancer reports it healthy. Unhealthy tasks are stopped and replaced. The health check grace period (healthCheckGracePeriodSeconds, default 0) tells the scheduler to ignore ALB and container health checks for that long after a task starts.
ECS container health check (in the task definition):
| Setting | Range | Default |
|---|---|---|
command | CMD or CMD-SHELL array; exit 0 = healthy | — |
interval | 5–300 s | 30 s |
timeout | 2–60 s | 5 s |
retries | 1–10 | 3 |
startPeriod | 0–300 s | off |
If an essential container becomes unhealthy, the task is stopped and the service replaces it. The command runs inside the container, so it needs a binary to run — a problem for chiseled .NET images, which have no shell or curl.
Elastic Beanstalk: load-balanced environments use the ALB health check path (aws:elasticbeanstalk:environment:process:default → HealthCheckPath). Enhanced health reporting combines that with an on-instance agent's data (request rates, 5xx percentages, latency, CPU) into a colour (Green/Yellow/Red/Grey). Single-instance environments have no ALB health check.
Lambda and API Gateway have no health checks: each invocation either succeeds or fails, and you monitor error rates and throttles with CloudWatch alarms. For multi-Region failover, Route 53 health checks probe a public endpoint ⚠ covered in Module 12.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
ASP.NET Core AddHealthChecks() / MapHealthChecks() | Same | Identical code; AWS components just call the URLs |
| Kubernetes/AKS liveness, readiness, startup probes | ECS container health check (liveness), ALB health check (readiness), grace periods (start-up) | ECS has no separate readiness probe; ALB health also drives task replacement |
| Azure App Service Health check (instance removed, then replaced after an hour) | ALB health check + ECS service replacement / Beanstalk enhanced health | ECS replaces quickly — minutes, not an hour |
| Application Gateway health probes | ALB target group health checks | Thresholds and success codes are configurable per target group |
| Container Apps health probes | ECS container healthCheck | Runs a command inside the container, not an HTTP probe from outside |
| Azure Front Door / Traffic Manager probes | Route 53 health checks | Global DNS failover |
Architecture
Timing — how long until something happens?
| Event | Formula | Defaults (30 s, 5 healthy, 2 unhealthy) | Lab settings (15 s, 2 healthy, 3 unhealthy) |
|---|---|---|---|
| Healthy target stops getting traffic after it breaks | interval × unhealthy threshold | about 60 s | about 45 s |
| New target starts receiving traffic | interval × healthy threshold (after start-up) | about 150 s | about 30 s |
| ECS replaces a broken task | detection + task stop + new task start + healthy threshold | several minutes | 1–2 minutes |
The defaults are slow to bring new tasks into service (five successes at 30 seconds). That slows every deployment and every scale-out. Most .NET APIs use a shorter interval and a healthy threshold of 2 or 3.
Security-boundary view: health endpoints are anonymous (load balancers do not send tokens) and reachable by anyone who can reach the ALB, unless you block them at the listener. So:
- Return a minimal body: status per check, no exception messages, host names, connection strings or versions.
- Expose detailed dependency information only on an internal endpoint (different port or path blocked by an ALB listener rule), or only in logs and metrics.
- Keep checks cheap and rate-independent so a flood of requests to
/healthcannot become a denial-of-service on your database.
The restart-loop trap, step by step
How to avoid it:
- Liveness never checks dependencies. It answers "is this process able to answer an HTTP request at all?"
- Readiness checks only what this instance can fix by being replaced or what makes this instance useless: start-up complete, configuration loaded and valid, local caches warmed, the process not shutting down.
- Shared dependencies report
Degraded, notUnhealthy, on the routed endpoint. ASP.NET Core mapsDegradedto HTTP 200 by default, so the ALB keeps routing (the API can often still serve cached data or return a clean 503 ProblemDetails per request), while your metrics and logs show the degradation. - Cache dependency results for a few seconds so health traffic never hammers the database.
- Alert on dependency health through CloudWatch alarms for humans, and let circuit breakers and retries in the request path handle transient downstream failures.
The module lab's /health/ready checks DynamoDB DescribeTable and returns Unhealthy if it fails. DynamoDB is a regional, multi-AZ service, so a table-level outage affecting only your tasks is rare, and the lab keeps the check simple. For a relational database — or any dependency with planned failovers — use the pattern in this lesson's complete code instead.
Step-by-step implementation
Separate the endpoints
/health/live (no dependencies), /health/ready (start-up gate, shutdown flag, local checks; shared dependencies as Degraded) and /health/deps (deep check, internal only).
Add a start-up gate
A hosted service warms caches and flips a StartupGate to ready. Until then /health/ready returns 503 so the ALB does not send traffic to a cold instance.
Flip readiness off on shutdown
On ApplicationStopping, mark the instance not ready. Combined with the deregistration delay, in-flight requests finish and new ones go elsewhere.
Configure the ALB target group
Path /health/ready, interval 15 s, timeout 5 s, healthy threshold 2, unhealthy threshold 3, matcher 200, deregistration delay 30 s.
Configure ECS
Health check grace period a little longer than your worst-case start-up (measure it). Optionally a container health check on /health/live using an in-image probe, with a startPeriod.
Test the behaviour
Use WebApplicationFactory tests to prove: live is 200 even when the database is down; ready is 503 during start-up and shutdown; ready is 200 (Degraded) when the database is down; bodies contain no exception text.
AWS Console instructions
- Console → EC2 → Target Groups → your target group → Health checks → Edit. Protocol HTTP, path
/health/ready. Advanced health check settings: port Traffic port, healthy threshold 2, unhealthy threshold 3, timeout 5, interval 15, success codes 200. Save. - Attributes → Edit: deregistration delay 30 seconds.
- Targets tab: each target's Health status and Health status details (reason codes such as
Target.ResponseCodeMismatchorTarget.Timeout). - Console → Elastic Container Service → Clusters → cluster → Services → service → Update service: under Load balancing, set Health check grace period. Under Deployment options, keep the circuit breaker on.
- Task definitions → Create new revision → Container → HealthCheck: command
CMD, dotnet, /app/Harbourline.HealthDemo.dll, --probe, http://127.0.0.1:8080/health/live, interval 30, timeout 5, retries 3, start period 30. - Elastic Beanstalk → Environment → Configuration → Instance traffic and scaling → Processes → default → Edit: health check path
/health/ready. - Elastic Beanstalk → Environment → Health: per-instance status and causes from enhanced health.
AWS CLI
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
# ALB target group: readiness path and timings.
aws elbv2 modify-target-group --target-group-arn "$TG_ARN" \
--health-check-path /health/ready --health-check-interval-seconds 15 \
--health-check-timeout-seconds 5 --healthy-threshold-count 2 --unhealthy-threshold-count 3 \
--matcher HttpCode=200
aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" \
--attributes Key=deregistration_delay.timeout_seconds,Value=30
# Why is a target unhealthy? (reason codes: Target.Timeout, Target.ResponseCodeMismatch, ...)
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
--query 'TargetHealthDescriptions[].{ip:Target.Id,state:TargetHealth.State,reason:TargetHealth.Reason,desc:TargetHealth.Description}' \
--output table
# ECS: grace period for slow starters (seconds).
aws ecs update-service --cluster hl-m03-cluster --service hl-m03-accounts-api \
--health-check-grace-period-seconds 60
# ECS: container health status of running tasks.
aws ecs describe-tasks --cluster hl-m03-cluster \
--tasks $(aws ecs list-tasks --cluster hl-m03-cluster --service-name hl-m03-accounts-api --query 'taskArns' --output text) \
--query 'tasks[].{task:taskArn,health:healthStatus,containers:containers[].{name:name,health:healthStatus}}'
# ECS: why were tasks replaced? Look for "unhealthy" in service events.
aws ecs describe-services --cluster hl-m03-cluster --services hl-m03-accounts-api \
--query 'services[0].events[:10].[createdAt,message]' --output table
# Beanstalk: health with causes.
aws elasticbeanstalk describe-instances-health --environment-name hl-m03-accounts-dev \
--attribute-names HealthStatus Color Causes --query 'InstanceHealthList[].[InstanceId,HealthStatus,Causes]'Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.ECS;
using HealthCheck = Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck;
// Container health check: liveness via the in-image probe (no shell needed).
var container = taskDefinition.AddContainer("api", new ContainerDefinitionOptions
{
Image = ContainerImage.FromEcrRepository(repository, "v1"),
PortMappings = [new PortMapping { ContainerPort = 8080 }],
HealthCheck = new Amazon.CDK.AWS.ECS.HealthCheck
{
Command = ["CMD", "dotnet", "/app/Harbourline.HealthDemo.dll", "--probe", "http://127.0.0.1:8080/health/live"],
Interval = Duration.Seconds(30),
Timeout = Duration.Seconds(5),
Retries = 3,
StartPeriod = Duration.Seconds(30),
},
StopTimeout = Duration.Seconds(30),
});
// ALB target group: readiness.
service.TargetGroup.ConfigureHealthCheck(new HealthCheck
{
Path = "/health/ready",
Interval = Duration.Seconds(15),
Timeout = Duration.Seconds(5),
HealthyThresholdCount = 2,
UnhealthyThresholdCount = 3,
HealthyHttpCodes = "200",
});
service.TargetGroup.SetAttribute("deregistration_delay.timeout_seconds", "30");
// On the service props: HealthCheckGracePeriod = Duration.Seconds(60)The lab's FargateStack.cs uses the same ALB settings and a 60-second grace period, and omits the container health check (the comment in its Dockerfile explains why).
Complete code
A self-contained Minimal API that implements the full pattern. The database dependency is simulated by IAccountStore so you can run and test it without AWS.
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<RootNamespace>Harbourline.HealthDemo</RootNamespace>
</PropertyGroup>
</Project>using System.Text.Json;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;
namespace Harbourline.HealthDemo;
/// <summary>Stand-in for the real data store (DynamoDB, PostgreSQL ...).</summary>
public interface IAccountStore
{
Task<bool> PingAsync(CancellationToken cancellationToken);
}
/// <summary>Flips to ready when warm-up completes, and back to not-ready when shutdown starts.</summary>
public sealed class StartupGate
{
private volatile bool _ready;
private volatile bool _stopping;
public bool IsReady => _ready && !_stopping;
public void MarkReady() => _ready = true;
public void MarkStopping() => _stopping = true;
}
/// <summary>Readiness: has start-up finished, and are we not shutting down? Instance-local only.</summary>
public sealed class StartupGateCheck(StartupGate gate) : IHealthCheck
{
public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) =>
Task.FromResult(gate.IsReady
? HealthCheckResult.Healthy()
: HealthCheckResult.Unhealthy("Starting up or shutting down."));
}
/// <summary>
/// Shared dependency: reports Degraded (HTTP 200) on failure so the orchestrator does not replace
/// healthy tasks for an outage they cannot fix. Results are cached so health traffic stays cheap.
/// </summary>
public sealed class AccountStoreCheck(IAccountStore store, TimeProvider clock) : IHealthCheck
{
private static readonly TimeSpan CacheFor = TimeSpan.FromSeconds(10);
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(2);
private readonly SemaphoreSlim _lock = new(1, 1);
private (DateTimeOffset At, bool Ok)? _cached; // cache the raw outcome; each registration maps it to its own status
public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) =>
await IsUpAsync(cancellationToken)
? HealthCheckResult.Healthy()
: new HealthCheckResult(context.Registration.FailureStatus, "Account store unreachable.");
private async Task<bool> IsUpAsync(CancellationToken cancellationToken)
{
if (_cached is { } c && clock.GetUtcNow() - c.At < CacheFor) return c.Ok;
await _lock.WaitAsync(cancellationToken);
try
{
if (_cached is { } again && clock.GetUtcNow() - again.At < CacheFor) return again.Ok;
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
cts.CancelAfter(Timeout);
bool ok;
try { ok = await store.PingAsync(cts.Token); }
catch (Exception) { ok = false; } // never surface exception text in a health response
_cached = (clock.GetUtcNow(), ok);
return ok;
}
finally
{
_lock.Release();
}
}
}
/// <summary>Warms up (e.g. loads reference data) and then opens the gate.</summary>
public sealed class WarmUpService(StartupGate gate, IHostApplicationLifetime lifetime, ILogger<WarmUpService> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
lifetime.ApplicationStopping.Register(() =>
{
gate.MarkStopping(); // readiness → 503 immediately; the ALB stops sending new requests
logger.LogInformation("Shutdown started: readiness now failing.");
});
await Task.Delay(TimeSpan.FromSeconds(3), stoppingToken); // stand-in for cache warm-up
gate.MarkReady();
logger.LogInformation("Warm-up complete: ready for traffic.");
}
}
public static class HealthEndpoints
{
public const string Live = "live", Ready = "ready", Deps = "deps";
public static IServiceCollection AddHarbourlineHealth(this IServiceCollection services)
{
services.AddSingleton<StartupGate>();
services.AddSingleton(TimeProvider.System);
services.AddHostedService<WarmUpService>();
// Singleton so the cache survives between probes (AddCheck<T> resolves T from DI if registered).
services.AddSingleton<AccountStoreCheck>();
services.AddHealthChecks()
.AddCheck("self", () => HealthCheckResult.Healthy(), tags: [Live])
.AddCheck<StartupGateCheck>("startup", tags: [Ready])
// On the routed endpoint a store failure is Degraded (200); on /health/deps it is Unhealthy.
.AddCheck<AccountStoreCheck>("account-store", failureStatus: HealthStatus.Degraded, tags: [Ready])
.AddCheck<AccountStoreCheck>("account-store-deep", failureStatus: HealthStatus.Unhealthy, tags: [Deps]);
return services;
}
public static WebApplication MapHarbourlineHealth(this WebApplication app)
{
app.MapHealthChecks("/health/live", Options(Live));
app.MapHealthChecks("/health/ready", Options(Ready));
// Internal only: block /health/deps at the ALB listener (or bind it to a separate port).
app.MapHealthChecks("/health/deps", Options(Deps));
return app;
}
private static HealthCheckOptions Options(string tag) => new()
{
Predicate = r => r.Tags.Contains(tag),
ResponseWriter = WriteMinimalAsync,
// Defaults: Healthy → 200, Degraded → 200, Unhealthy → 503. Stated explicitly for reviewers.
ResultStatusCodes =
{
[HealthStatus.Healthy] = StatusCodes.Status200OK,
[HealthStatus.Degraded] = StatusCodes.Status200OK,
[HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable,
},
};
// Status per check only: no descriptions, exceptions, host names or versions.
private static Task WriteMinimalAsync(HttpContext context, HealthReport report)
{
context.Response.ContentType = "application/json";
context.Response.Headers.CacheControl = "no-store";
return context.Response.WriteAsync(JsonSerializer.Serialize(new
{
status = report.Status.ToString(),
checks = report.Entries.ToDictionary(e => e.Key, e => e.Value.Status.ToString()),
}));
}
}using Harbourline.HealthDemo;
// In-image probe for the ECS container health check: chiseled images have no curl or shell,
// so the app itself can act as the probe. Exit code 0 = healthy, 1 = unhealthy.
if (args is ["--probe", var probeUrl])
{
using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(3) };
try
{
using var response = await http.GetAsync(probeUrl);
return response.IsSuccessStatusCode ? 0 : 1;
}
catch
{
return 1;
}
}
var builder = WebApplication.CreateBuilder(args);
// Graceful shutdown budget: keep it at or below the ECS stopTimeout (30 s default, 120 s max on Fargate).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));
builder.Services.AddSingleton<IAccountStore, SimulatedAccountStore>();
builder.Services.AddHarbourlineHealth();
var app = builder.Build();
app.MapHarbourlineHealth();
app.MapGet("/accounts/{id}", (string id) => Results.Ok(new { accountId = id, currency = "GBP" }));
// Lab helper to simulate a database outage. Development only.
if (app.Environment.IsDevelopment())
{
app.MapPost("/dev/store/{up:bool}", (bool up, IAccountStore store) =>
{
((SimulatedAccountStore)store).Up = up;
return Results.NoContent();
});
}
await app.RunAsync();
return 0;
public sealed class SimulatedAccountStore : IAccountStore
{
public volatile bool Up = true;
public Task<bool> PingAsync(CancellationToken cancellationToken) => Task.FromResult(Up);
}
public partial class Program;using System.Net;
using Harbourline.HealthDemo;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;
using Xunit;
public class HealthTests(WebApplicationFactory<Program> factory) : IClassFixture<WebApplicationFactory<Program>>
{
private async Task<HttpClient> ReadyClientAsync()
{
var client = factory.CreateClient();
factory.Services.GetRequiredService<StartupGate>().MarkReady(); // skip the 3 s warm-up in tests
await Task.CompletedTask;
return client;
}
private SimulatedAccountStore Store => (SimulatedAccountStore)factory.Services.GetRequiredService<IAccountStore>();
[Fact]
public async Task Live_is_200_even_when_the_store_is_down()
{
var client = await ReadyClientAsync();
Store.Up = false;
try
{
var response = await client.GetAsync("/health/live", TestContext.Current.CancellationToken);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
finally { Store.Up = true; }
}
[Fact]
public async Task Ready_reports_degraded_but_200_when_the_shared_store_is_down()
{
var client = await ReadyClientAsync();
Store.Up = false;
try
{
await Task.Delay(TimeSpan.FromSeconds(11), TestContext.Current.CancellationToken); // let any cached result expire
var response = await client.GetAsync("/health/ready", TestContext.Current.CancellationToken);
var body = await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
Assert.Contains("Degraded", body);
Assert.DoesNotContain("Exception", body);
}
finally { Store.Up = true; }
}
[Fact]
public async Task Ready_is_503_while_stopping()
{
// Own factory: closing the gate is permanent, so it must not affect the shared fixture.
await using var own = new WebApplicationFactory<Program>();
var client = own.CreateClient();
var gate = own.Services.GetRequiredService<StartupGate>();
gate.MarkReady();
gate.MarkStopping();
var response = await client.GetAsync("/health/ready", TestContext.Current.CancellationToken);
Assert.Equal(HttpStatusCode.ServiceUnavailable, response.StatusCode);
}
}The test project follows CONVENTIONS.md: xunit.v3 4.0.3, Microsoft.AspNetCore.Mvc.Testing 10.0.15, <OutputType>Exe</OutputType>, Microsoft.Testing.Platform runner. The 11-second wait in the Degraded test lets the 10-second dependency cache expire; inject a fake TimeProvider (from Microsoft.Extensions.TimeProvider.Testing) to make it instant.
Code explained line by line
StartupGate— twovolatileflags read by the readiness check on every probe. Volatile because they are written by one thread (hosted service, shutdown callback) and read by request threads.StartupGateCheck— the core of readiness:Unhealthy(503) before warm-up finishes and after shutdown begins. These are instance-local states that the ALB should act on.AccountStoreCheck— the dependency check:- caches its result for 10 seconds, so 3 ALB nodes × N tasks × every 15 seconds never become a meaningful load on the database;
- uses a 2-second timeout so a hanging dependency cannot make the health endpoint exceed the ALB's 5-second timeout;
- swallows exception details, because health responses are anonymous;
- returns
context.Registration.FailureStatus, so the same class isDegradedon/health/readyandUnhealthyon/health/deps.
WarmUpService— aBackgroundServicethat opens the gate when warm-up completes, and registers anApplicationStoppingcallback that closes it as soon asSIGTERMarrives.AddCheck(..., tags: [...])+Predicate— tags select which checks run on which endpoint. Liveness runs onlyself, which can never fail while the process can serve HTTP.ResultStatusCodes— the ASP.NET Core defaults (Degraded→ 200) written out, so a reviewer sees the decision.WriteMinimalAsync— status names only, plusCache-Control: no-storeso no proxy caches a health result.args is ["--probe", var probeUrl]— a list pattern: when the container health check runsdotnet app.dll --probe http://127.0.0.1:8080/health/live, the app makes one HTTP call and exits with 0 or 1, without starting the web host. Cost: each probe starts a short-lived .NET process (tens of MB for a fraction of a second); on a 512 MB task with a 30-second interval this is acceptable, but measure. The alternative is no container health check at all, relying on the ALB.HostOptions.ShutdownTimeout = 25 s— how long the host waits for in-flight requests and hosted services afterSIGTERM. Keep it below ECSstopTimeout(30 s by default) so the process exits cleanly beforeSIGKILL./dev/store/{up}— a Development-only switch to simulate an outage locally and watch the endpoints change.- Tests use
WebApplicationFactory<Program>— the real pipeline in memory. They prove the behaviour that matters: liveness ignores dependencies, readiness degrades instead of failing for a shared dependency, and readiness fails during shutdown.
What graceful shutdown looks like on ECS
- ECS decides to stop a task (deployment, scale-in, replacement).
- ECS deregisters the task from the target group; the ALB stops sending new requests and keeps existing connections for up to the deregistration delay (30 s in our settings).
- ECS sends
SIGTERMto the container. .NET raisesApplicationStopping; our gate closes; Kestrel stops accepting new connections and waits for in-flight requests up toShutdownTimeout. - If the process has not exited after
stopTimeout, ECS sendsSIGKILL.
Keep: request timeout ≤ deregistration delay, and ShutdownTimeout < stopTimeout.
Security considerations
⚠ Learning shortcut — not for production
The demo maps /health/deps on the same port as the public API and includes a Development-only endpoint that toggles the simulated store. In production: serve deep checks on an internal-only listener or block the path with an ALB listener rule (fixed 404 response for /health/deps*), remove all test switches, and publish dependency status as CloudWatch metrics for dashboards instead of an HTTP endpoint.
- Health endpoints are anonymous by design — treat their output as public. No stack traces, no exception messages, no versions or build numbers that help an attacker pick exploits.
- Keep them cheap and rate-independent (cached checks, timeouts) so they cannot be used to amplify load on your database.
- Exclude health probes from request logs at Information level (the lab logs them only at Verbose), otherwise probes dominate log volume and cost.
- Do not put health endpoints behind authentication middleware by accident:
.AllowAnonymous()(as the lab does) when a fallback authorisation policy is configured. - A health check that makes outbound calls needs the same least-privilege permissions as the app; the lab's DynamoDB check needs
dynamodb:DescribeTable, which the task role grants explicitly.
Banking use case
After the Monday incident described at the start, Harbourline's platform team publishes a health check standard for all ECS services:
/health/live: process only. Used by the container health check where an in-image probe exists; otherwise omitted./health/ready: start-up gate, configuration validated (thanks toValidateOnStart), shutdown flag; shared dependencies (PostgreSQL, the core-banking adapter, the sanctions service) asDegradedwith cached checks./health/deps: blocked at the ALB, scraped by a small internal job that publishes per-dependency CloudWatch metrics; alarms page the owning team.- Target group: 15/5/2/3, matcher 200, deregistration delay matched to the longest request timeout (30 s). Grace period: measured p99 start-up + 50%.
The next primary database failover produced 40 seconds of 503 ProblemDetails responses for writes — as designed — with zero task replacements, and the incident report took 10 minutes to write instead of a day.
Common mistakes
- Liveness checks the database → every task killed during a database blip → liveness checks the process only.
- Readiness
Unhealthyfor a shared dependency on ECS → restart loop →Degraded, alarms, and request-level resilience. - Health check path requires authentication → every target unhealthy →
AllowAnonymous(). - Grace period shorter than start-up → tasks killed before they finish starting → measure start-up and add margin.
- Grace period of 10 minutes "to be safe" → genuinely broken tasks keep failing for 10 minutes → set it from measurements.
- Default healthy threshold (5 × 30 s) → deployments and scale-out take minutes longer → 2–3 × 15 s.
HEALTHCHECK CMD curl ...in a chiseled image → container always unhealthy becausecurldoes not exist → in-image probe or no container check.- Returning exception details → internal hostnames leaked → minimal response writer.
ShutdownTimeoutlonger thanstopTimeout→SIGKILLmid-request → keep it shorter.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Targets unhealthy, reason Target.ResponseCodeMismatch | Endpoint returns 401/404/503 | Check path, authentication and readiness logic; curl from inside the VPC |
Targets unhealthy, reason Target.Timeout | Security group blocks ALB → task, wrong port, or slow check | Allow the ALB security group on the container port; add a timeout and caching to checks |
| Tasks replaced every few minutes | Liveness/readiness fails on a dependency, or grace period too short | Read service events; separate checks; raise the grace period from measurements |
Container UNHEALTHY but ALB says healthy | Container health command fails (missing binary, wrong URL, IPv6 localhost) | Use 127.0.0.1, an in-image probe, and check the command with docker run locally |
| 502s during every deployment | App killed before draining; deregistration delay too short for long requests | Readiness off on ApplicationStopping; delay ≥ request timeout; ShutdownTimeout < stopTimeout |
| Beanstalk health Red: "ELB health is failing" | Beanstalk process health check path wrong or authenticated | Set HealthCheckPath; make it anonymous |
| All targets unhealthy but traffic still flows | ALB fail-open behaviour | Fix the check; do not rely on fail-open |
| Health requests dominate logs | Probes logged at Information | Lower the log level for /health paths |
Cost and cleanup
What this lesson costs: the code and tests run locally for free. If you apply the settings to the Fargate or Beanstalk environment from the earlier lessons, the costs are those lessons' costs; health checks themselves are free (ALB health checks do not consume LCUs ⚠ VERIFY). The container health check uses a small amount of task CPU and memory. Label: free.
Cleanup: nothing new is created in AWS by this lesson. If you deployed the Fargate lesson's stack to test these settings, follow its teardown, then confirm:
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: Run the demo locally. Call
/health/live,/health/readyand/health/depsduring the first 3 seconds and after. ThenPOST /dev/store/falseand call them again after 10 seconds. Write down each status code and explain it. - 🟡 Challenge: Compute detection and recovery times for three target-group configurations (defaults; 15/5/2/3; 5/2/2/2) and a 20-second start-up. Which configuration risks false failures during garbage-collection pauses? Then add a fourth test proving that
/health/readyreturns 503 during the first second after start-up. - 🔴 Live problem: Ticket INC-6018: "During the 02:00 database maintenance window last night, the Statements API (ECS, 6 tasks) cycled through 41 task replacements, and the deployment circuit breaker marked this morning's release as failed even though it was fine." Review the attached task definition: container health check
CMD-SHELL curl -f http://localhost:8080/health || exit 1, ALB path/health(which runsSELECT 1on PostgreSQL), grace period 0. Produce a root-cause analysis and corrected settings.
Assignment
Add a HEALTH.md to your portfolio repository: your service's three endpoints and what each checks, the AWS settings (target group, grace period, container check, deregistration delay, stop timeout, shutdown timeout) with the reasoning and a timing table, and the WebApplicationFactory tests from this lesson adapted to your API.
Solutions
First 3 s: live 200; ready 503 (gate closed); deps 200 (store up). After warm-up: all 200, ready body Healthy. After POST /dev/store/false and 10 s (cache expiry): live 200 (no dependencies); ready 200 with status Degraded and account-store: Degraded; deps 503 with Unhealthy. Why: the same check registered twice with different failure statuses gives the router a "keep sending traffic" signal and monitoring a "something is wrong" signal.
Defaults (30/5/5/2): detect about 60 s; new task in service about 20 s start-up + 150 s = 170 s. 15/5/2/3: detect about 45 s; in service about 20 + 30 = 50 s. 5/2/2/2: detect about 10 s; in service about 30 s — but a 2-second timeout and only two failures mean a couple of slow responses during a full GC or a CPU spike can remove a healthy task: risk of false failures and flapping. For the extra test, create a fresh WebApplicationFactory, request /health/ready immediately without calling MarkReady, and assert 503.
Root cause: (1) The ALB path /health runs SELECT 1, so during maintenance every task was marked unhealthy; ECS replaces tasks the ALB reports unhealthy, so all six were replaced repeatedly (41 replacements). (2) The container health check uses curl — if the image is chiseled, it always fails; if not, it also hit the database-dependent path, doubling the effect. (3) Grace period 0 meant new tasks were judged before they finished starting, adding more failures. (4) The circuit breaker counts failed task launches; the release deployed during the churn accumulated failures and was rolled back although the code was fine. Corrected settings: /health/live (process only) for the container check via an in-image probe (or no container check); ALB path /health/ready with start-up gate and the database reported as Degraded; deep check on an internal path feeding a CloudWatch alarm; grace period = measured start-up × 1.5; target group 15/5/2/3. Process: suppress deployments during maintenance windows and add the window to the change calendar.
Knowledge check
Answer: The ALB stops routing new requests to it, and the ECS service scheduler treats it as unhealthy (after any grace period) and replaces it.
Answer: Restarting the process does not fix the database. A dependency failure would make the orchestrator kill every healthy instance, turning a dependency blip into a full outage (the restart loop).
Answer: 200. Healthy → 200, Degraded → 200, Unhealthy → 503, configurable through ResultStatusCodes.
Answer: For that many seconds after a task starts, the scheduler ignores ALB and container health check failures, giving the application time to start. The default is 0.
Answer: It fails open: it routes requests to all targets regardless of health status.
Interview questions
- "Explain liveness vs readiness on AWS." — Liveness: is the process alive? Used by the ECS container health check; failure means replacement. Readiness: should it get traffic? Used by the ALB; on ECS, failure also leads to replacement, which is why readiness must not fail for shared dependencies.
- "How would you tune health checks for a .NET API on Fargate?" — Measure start-up; grace period from measurements; ALB 15 s interval, 5 s timeout, 2 healthy, 3 unhealthy; deregistration delay ≥ longest request;
ShutdownTimeoutbelowstopTimeout; cheap, cached checks with minimal bodies. - "A dependency is down. What should the API do?" — Stay up, report
Degraded, fail fast per request with ProblemDetails (circuit breaker), serve cached data where safe, alert humans. Do not let the orchestrator restart everything. - "How do you do health checks with a distroless or chiseled image?" — No shell or curl: rely on the ALB check, or add an in-image probe mode (the app itself or a tiny native binary) for the container health check.
Certification practice
An ASP.NET Core API on ECS Fargate takes 50 seconds to start because it loads reference data. After each deployment, new tasks are stopped with "failed ELB health checks" before they finish starting, and the deployment eventually fails. The ALB health check is correct. What should the developer change?
A. Increase the ALB unhealthy threshold to 10.
B. Set the ECS service health check grace period to about 75 seconds.
C. Change the health check path to /.
D. Turn off the deployment circuit breaker.
Answer: B. The grace period makes ECS ignore health check failures while the task starts. A only partially delays detection and slows real failure detection everywhere. C does not fix start-up time and may return 404. D hides the symptom and allows broken deployments.
During rolling deployments of an API behind an ALB, clients see a burst of 502 errors when old tasks stop. Requests take up to 10 seconds. Which combination of changes reduces these errors? (Choose TWO.)
A. Make the application report not-ready as soon as it receives SIGTERM and finish in-flight requests before exiting.
B. Set the target group deregistration delay to at least the longest request duration.
C. Set the deregistration delay to 0 seconds.
D. Remove the ALB health check.
E. Set minimumHealthyPercent to 0.
Answer: A and B. Graceful shutdown plus an adequate deregistration delay let in-flight requests complete while new ones go to other targets. C cuts connections immediately. D removes protection. E allows ECS to stop all old tasks before new ones are ready.
A company's web tier runs on ECS behind an ALB. Its health check endpoint verifies connectivity to an Aurora database. During a 30-second Aurora failover, all tasks were replaced and the application was unavailable for 8 minutes. Which design change best improves resilience?
A. Increase the number of tasks from 4 to 12. B. Make the load balancer health check verify only that the application instance can serve requests, and monitor database connectivity separately with CloudWatch alarms. C. Point the health check at the Aurora reader endpoint. D. Move the web tier to a single larger task.
Answer: B. Health checks that depend on a shared database make every instance fail together and trigger mass replacement. A adds more tasks that would all fail the same way. C still couples instance health to the database. D removes redundancy.
Completion checklist
- I can explain liveness, readiness, start-up and dependency checks and who calls each
- I ran the demo and saw
Degradedkeep readiness at 200 while the store was down - I can calculate detection and recovery time from target-group settings
- I know the shutdown sequence on ECS and how to configure it
- I wrote (or ran) the
WebApplicationFactoryhealth tests
Key takeaways
- Only fail a check if the resulting action would fix the problem. Restarts fix stuck processes, not databases.
- On ECS behind an ALB, ALB-unhealthy means replaced — so readiness must not fail for shared dependencies; use
Degradedand alarms. - Tune timings deliberately: grace period from measured start-up, 15 s interval, low healthy threshold, short deregistration delay matched to request timeouts.
- Close readiness on
SIGTERM, keepShutdownTimeoutbelowstopTimeout, and let the ALB drain. - Health endpoints are anonymous and public: minimal bodies, cached checks, no secrets or exception text.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Secrets in .NET on AWS
Store credentials and keys in AWS Secrets Manager, read them from .NET with client-side caching, survive rotation without outages, write a C# rotation function, and know why environment variables are the wrong place for secrets.
Lab 3.2: containerise the Accounts API with a chiseled, non-root image
Build the module project's ASP.NET Core API into a multi-stage, chiseled, non-root container image, run it locally with docker compose under production-like restrictions, and check its size, user and start-up time.