Skip to Content
Cloud Engineering Academy
3. .NET Application Development on AWS

Health checks that help instead of hurt

Design liveness and readiness endpoints in ASP.NET Core, wire them correctly to ALB target groups, ECS container health checks and Elastic Beanstalk, tune timings and grace periods, shut down gracefully, and avoid the restart loop that turns a database blip into a full outage.

intermediate⏱ 1 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · ASP.NET Core health checks · Microsoft.AspNetCore.Mvc.Testing 10.0.15 · xunit.v3 4.0.3 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain the difference between liveness, readiness and start-up checks, and which AWS component should call each.
  • Implement health endpoints in ASP.NET Core with tags, predicates, safe response bodies and cached dependency checks.
  • Configure ALB target group health checks, ECS container health checks, the ECS health check grace period and Elastic Beanstalk health settings — and calculate how long detection and recovery take.
  • Recognise and prevent the restart-loop trap, where a shared dependency outage makes the orchestrator kill every healthy task.
  • Implement graceful shutdown so deployments and scale-in do not drop in-flight requests.
  • Test health behaviour with WebApplicationFactory and xUnit, without AWS.

Prerequisites

Why this matters in banking

At 09:02 on a Monday, Harbourline's PostgreSQL primary fails over to its standby. The failover takes 40 seconds — well within the database's design. But the Accounts API's health endpoint runs SELECT 1 against the database, and the load balancer is configured to check it every 10 seconds with an unhealthy threshold of 2. Within 20 seconds every task is marked unhealthy. ECS, doing exactly what it was told, stops every task and starts new ones. The new tasks take 25 seconds to start, then also fail their health checks because the database is still failing over. By the time the database is back, ECS is in a replacement loop, the deployment circuit breaker has tripped on an unrelated release, and the 40-second database event has become an 11-minute customer-facing outage on the mobile app.

The reverse also happens: an endpoint that always returns 200 keeps sending traffic to a task whose thread pool is exhausted, so 25% of requests time out for an hour.

Health checks are small pieces of code with outsized influence. Operational-resilience reviews (and regulators' expectations that banks stay within impact tolerances for important business services) make "how does the platform decide an instance is broken?" a real question with a documented answer.

The idea in plain language

A health check is a question the platform asks your application on a schedule. The trick is that there are different questions, asked by different components, with different consequences:

CheckQuestionWho asksConsequence of "no"
Liveness"Is this process alive and not stuck?"Container orchestrator (ECS container health check)The container/task is killed and replaced
Readiness"Should this instance receive traffic right now?"Load balancer (ALB target group)Traffic is routed elsewhere — but on ECS, a task the ALB marks unhealthy is also replaced
Start-up"Have you finished starting?"Grace periods (ECS healthCheckGracePeriodSeconds, container startPeriod)Failures are ignored until start-up ends
Dependency / deep check"Are my downstream systems OK?"Monitoring, dashboards, alarmsA human or an alarm decides — not automatic replacement

The golden rule: only fail a check whose consequence would actually fix the problem. Restarting a task fixes a deadlocked process. Restarting a task does not fix a database outage — every replacement connects to the same database.

Analogy: A bank branch has a receptionist (load balancer) and a regional manager (orchestrator). The receptionist asks each cashier "can you take a customer?" — if a cashier is on a break, customers go to someone else (readiness). The regional manager asks "are you actually at work?" — if a cashier is missing, the manager sends a replacement (liveness). If the card-payment network is down, the receptionist should still send customers to cashiers who can help with other things, and the manager should certainly not fire and replace every cashier: the network is not their fault.

What problem AWS health checks solve — and how each one behaves

Application Load Balancer target group health checks (checked 2026-11-18):

SettingRangeDefault (ip/instance targets)
HealthCheckIntervalSeconds5–30030
HealthCheckTimeoutSeconds2–1205
HealthyThresholdCount2–105
UnhealthyThresholdCount2–102
Matcher (success codes)200–499200
HealthCheckPathany/
  • A new target must pass health checks before receiving traffic.
  • Fail-open: if all targets in a target group are unhealthy, the ALB routes to all of them anyway. This protects you from a bad check taking everything offline at the routing layer — but not from ECS replacing the tasks.
  • Each ALB node checks each target independently, so with several AZs a task receives several health requests per interval. Keep the endpoint cheap.
  • Deregistration delay (default 300 seconds) is how long the ALB keeps in-flight connections to a deregistering target.

ECS service behaviour with a load balancer: a task is healthy only if it is RUNNING and the load balancer reports it healthy. Unhealthy tasks are stopped and replaced. The health check grace period (healthCheckGracePeriodSeconds, default 0) tells the scheduler to ignore ALB and container health checks for that long after a task starts.

ECS container health check (in the task definition):

SettingRangeDefault
commandCMD or CMD-SHELL array; exit 0 = healthy—
interval5–300 s30 s
timeout2–60 s5 s
retries1–103
startPeriod0–300 soff

If an essential container becomes unhealthy, the task is stopped and the service replaces it. The command runs inside the container, so it needs a binary to run — a problem for chiseled .NET images, which have no shell or curl.

Elastic Beanstalk: load-balanced environments use the ALB health check path (aws:elasticbeanstalk:environment:process:default → HealthCheckPath). Enhanced health reporting combines that with an on-instance agent's data (request rates, 5xx percentages, latency, CPU) into a colour (Green/Yellow/Red/Grey). Single-instance environments have no ALB health check.

Lambda and API Gateway have no health checks: each invocation either succeeds or fails, and you monitor error rates and throttles with CloudWatch alarms. For multi-Region failover, Route 53 health checks probe a public endpoint ⚠ covered in Module 12.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
ASP.NET Core AddHealthChecks() / MapHealthChecks()SameIdentical code; AWS components just call the URLs
Kubernetes/AKS liveness, readiness, startup probesECS container health check (liveness), ALB health check (readiness), grace periods (start-up)ECS has no separate readiness probe; ALB health also drives task replacement
Azure App Service Health check (instance removed, then replaced after an hour)ALB health check + ECS service replacement / Beanstalk enhanced healthECS replaces quickly — minutes, not an hour
Application Gateway health probesALB target group health checksThresholds and success codes are configurable per target group
Container Apps health probesECS container healthCheckRuns a command inside the container, not an HTTP probe from outside
Azure Front Door / Traffic Manager probesRoute 53 health checksGlobal DNS failover

Architecture

Timing — how long until something happens?

EventFormulaDefaults (30 s, 5 healthy, 2 unhealthy)Lab settings (15 s, 2 healthy, 3 unhealthy)
Healthy target stops getting traffic after it breaksinterval × unhealthy thresholdabout 60 sabout 45 s
New target starts receiving trafficinterval × healthy threshold (after start-up)about 150 sabout 30 s
ECS replaces a broken taskdetection + task stop + new task start + healthy thresholdseveral minutes1–2 minutes

The defaults are slow to bring new tasks into service (five successes at 30 seconds). That slows every deployment and every scale-out. Most .NET APIs use a shorter interval and a healthy threshold of 2 or 3.

Security-boundary view: health endpoints are anonymous (load balancers do not send tokens) and reachable by anyone who can reach the ALB, unless you block them at the listener. So:

  • Return a minimal body: status per check, no exception messages, host names, connection strings or versions.
  • Expose detailed dependency information only on an internal endpoint (different port or path blocked by an ALB listener rule), or only in logs and metrics.
  • Keep checks cheap and rate-independent so a flood of requests to /health cannot become a denial-of-service on your database.

The restart-loop trap, step by step

How to avoid it:

  1. Liveness never checks dependencies. It answers "is this process able to answer an HTTP request at all?"
  2. Readiness checks only what this instance can fix by being replaced or what makes this instance useless: start-up complete, configuration loaded and valid, local caches warmed, the process not shutting down.
  3. Shared dependencies report Degraded, not Unhealthy, on the routed endpoint. ASP.NET Core maps Degraded to HTTP 200 by default, so the ALB keeps routing (the API can often still serve cached data or return a clean 503 ProblemDetails per request), while your metrics and logs show the degradation.
  4. Cache dependency results for a few seconds so health traffic never hammers the database.
  5. Alert on dependency health through CloudWatch alarms for humans, and let circuit breakers and retries in the request path handle transient downstream failures.

The module lab's /health/ready checks DynamoDB DescribeTable and returns Unhealthy if it fails. DynamoDB is a regional, multi-AZ service, so a table-level outage affecting only your tasks is rare, and the lab keeps the check simple. For a relational database — or any dependency with planned failovers — use the pattern in this lesson's complete code instead.

Step-by-step implementation

Separate the endpoints

/health/live (no dependencies), /health/ready (start-up gate, shutdown flag, local checks; shared dependencies as Degraded) and /health/deps (deep check, internal only).

Add a start-up gate

A hosted service warms caches and flips a StartupGate to ready. Until then /health/ready returns 503 so the ALB does not send traffic to a cold instance.

Flip readiness off on shutdown

On ApplicationStopping, mark the instance not ready. Combined with the deregistration delay, in-flight requests finish and new ones go elsewhere.

Configure the ALB target group

Path /health/ready, interval 15 s, timeout 5 s, healthy threshold 2, unhealthy threshold 3, matcher 200, deregistration delay 30 s.

Configure ECS

Health check grace period a little longer than your worst-case start-up (measure it). Optionally a container health check on /health/live using an in-image probe, with a startPeriod.

Test the behaviour

Use WebApplicationFactory tests to prove: live is 200 even when the database is down; ready is 503 during start-up and shutdown; ready is 200 (Degraded) when the database is down; bodies contain no exception text.

AWS Console instructions

  1. Console → EC2 → Target Groups → your target group → Health checks → Edit. Protocol HTTP, path /health/ready. Advanced health check settings: port Traffic port, healthy threshold 2, unhealthy threshold 3, timeout 5, interval 15, success codes 200. Save.
  2. Attributes → Edit: deregistration delay 30 seconds.
  3. Targets tab: each target's Health status and Health status details (reason codes such as Target.ResponseCodeMismatch or Target.Timeout).
  4. Console → Elastic Container Service → Clusters → cluster → Services → service → Update service: under Load balancing, set Health check grace period. Under Deployment options, keep the circuit breaker on.
  5. Task definitions → Create new revision → Container → HealthCheck: command CMD, dotnet, /app/Harbourline.HealthDemo.dll, --probe, http://127.0.0.1:8080/health/live, interval 30, timeout 5, retries 3, start period 30.
  6. Elastic Beanstalk → Environment → Configuration → Instance traffic and scaling → Processes → default → Edit: health check path /health/ready.
  7. Elastic Beanstalk → Environment → Health: per-instance status and causes from enhanced health.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1

# ALB target group: readiness path and timings.
aws elbv2 modify-target-group --target-group-arn "$TG_ARN" \
  --health-check-path /health/ready --health-check-interval-seconds 15 \
  --health-check-timeout-seconds 5 --healthy-threshold-count 2 --unhealthy-threshold-count 3 \
  --matcher HttpCode=200
aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" \
  --attributes Key=deregistration_delay.timeout_seconds,Value=30

# Why is a target unhealthy? (reason codes: Target.Timeout, Target.ResponseCodeMismatch, ...)
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
  --query 'TargetHealthDescriptions[].{ip:Target.Id,state:TargetHealth.State,reason:TargetHealth.Reason,desc:TargetHealth.Description}' \
  --output table

# ECS: grace period for slow starters (seconds).
aws ecs update-service --cluster hl-m03-cluster --service hl-m03-accounts-api \
  --health-check-grace-period-seconds 60

# ECS: container health status of running tasks.
aws ecs describe-tasks --cluster hl-m03-cluster \
  --tasks $(aws ecs list-tasks --cluster hl-m03-cluster --service-name hl-m03-accounts-api --query 'taskArns' --output text) \
  --query 'tasks[].{task:taskArn,health:healthStatus,containers:containers[].{name:name,health:healthStatus}}'

# ECS: why were tasks replaced? Look for "unhealthy" in service events.
aws ecs describe-services --cluster hl-m03-cluster --services hl-m03-accounts-api \
  --query 'services[0].events[:10].[createdAt,message]' --output table

# Beanstalk: health with causes.
aws elasticbeanstalk describe-instances-health --environment-name hl-m03-accounts-dev \
  --attribute-names HealthStatus Color Causes --query 'InstanceHealthList[].[InstanceId,HealthStatus,Causes]'

Infrastructure as Code

Health settings (excerpt from the Fargate stack)
using Amazon.CDK;
using Amazon.CDK.AWS.ECS;
using HealthCheck = Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck;

// Container health check: liveness via the in-image probe (no shell needed).
var container = taskDefinition.AddContainer("api", new ContainerDefinitionOptions
{
    Image = ContainerImage.FromEcrRepository(repository, "v1"),
    PortMappings = [new PortMapping { ContainerPort = 8080 }],
    HealthCheck = new Amazon.CDK.AWS.ECS.HealthCheck
    {
        Command = ["CMD", "dotnet", "/app/Harbourline.HealthDemo.dll", "--probe", "http://127.0.0.1:8080/health/live"],
        Interval = Duration.Seconds(30),
        Timeout = Duration.Seconds(5),
        Retries = 3,
        StartPeriod = Duration.Seconds(30),
    },
    StopTimeout = Duration.Seconds(30),
});

// ALB target group: readiness.
service.TargetGroup.ConfigureHealthCheck(new HealthCheck
{
    Path = "/health/ready",
    Interval = Duration.Seconds(15),
    Timeout = Duration.Seconds(5),
    HealthyThresholdCount = 2,
    UnhealthyThresholdCount = 3,
    HealthyHttpCodes = "200",
});
service.TargetGroup.SetAttribute("deregistration_delay.timeout_seconds", "30");
// On the service props: HealthCheckGracePeriod = Duration.Seconds(60)

The lab's FargateStack.cs uses the same ALB settings and a 60-second grace period, and omits the container health check (the comment in its Dockerfile explains why).

Complete code

A self-contained Minimal API that implements the full pattern. The database dependency is simulated by IAccountStore so you can run and test it without AWS.

Harbourline.HealthDemo.csproj
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <RootNamespace>Harbourline.HealthDemo</RootNamespace>
  </PropertyGroup>
</Project>
Health.cs
using System.Text.Json;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;

namespace Harbourline.HealthDemo;

/// <summary>Stand-in for the real data store (DynamoDB, PostgreSQL ...).</summary>
public interface IAccountStore
{
    Task<bool> PingAsync(CancellationToken cancellationToken);
}

/// <summary>Flips to ready when warm-up completes, and back to not-ready when shutdown starts.</summary>
public sealed class StartupGate
{
    private volatile bool _ready;
    private volatile bool _stopping;
    public bool IsReady => _ready && !_stopping;
    public void MarkReady() => _ready = true;
    public void MarkStopping() => _stopping = true;
}

/// <summary>Readiness: has start-up finished, and are we not shutting down? Instance-local only.</summary>
public sealed class StartupGateCheck(StartupGate gate) : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) =>
        Task.FromResult(gate.IsReady
            ? HealthCheckResult.Healthy()
            : HealthCheckResult.Unhealthy("Starting up or shutting down."));
}

/// <summary>
/// Shared dependency: reports Degraded (HTTP 200) on failure so the orchestrator does not replace
/// healthy tasks for an outage they cannot fix. Results are cached so health traffic stays cheap.
/// </summary>
public sealed class AccountStoreCheck(IAccountStore store, TimeProvider clock) : IHealthCheck
{
    private static readonly TimeSpan CacheFor = TimeSpan.FromSeconds(10);
    private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(2);
    private readonly SemaphoreSlim _lock = new(1, 1);
    private (DateTimeOffset At, bool Ok)? _cached; // cache the raw outcome; each registration maps it to its own status

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) =>
        await IsUpAsync(cancellationToken)
            ? HealthCheckResult.Healthy()
            : new HealthCheckResult(context.Registration.FailureStatus, "Account store unreachable.");

    private async Task<bool> IsUpAsync(CancellationToken cancellationToken)
    {
        if (_cached is { } c && clock.GetUtcNow() - c.At < CacheFor) return c.Ok;

        await _lock.WaitAsync(cancellationToken);
        try
        {
            if (_cached is { } again && clock.GetUtcNow() - again.At < CacheFor) return again.Ok;

            using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
            cts.CancelAfter(Timeout);
            bool ok;
            try { ok = await store.PingAsync(cts.Token); }
            catch (Exception) { ok = false; } // never surface exception text in a health response

            _cached = (clock.GetUtcNow(), ok);
            return ok;
        }
        finally
        {
            _lock.Release();
        }
    }
}

/// <summary>Warms up (e.g. loads reference data) and then opens the gate.</summary>
public sealed class WarmUpService(StartupGate gate, IHostApplicationLifetime lifetime, ILogger<WarmUpService> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        lifetime.ApplicationStopping.Register(() =>
        {
            gate.MarkStopping(); // readiness → 503 immediately; the ALB stops sending new requests
            logger.LogInformation("Shutdown started: readiness now failing.");
        });

        await Task.Delay(TimeSpan.FromSeconds(3), stoppingToken); // stand-in for cache warm-up
        gate.MarkReady();
        logger.LogInformation("Warm-up complete: ready for traffic.");
    }
}

public static class HealthEndpoints
{
    public const string Live = "live", Ready = "ready", Deps = "deps";

    public static IServiceCollection AddHarbourlineHealth(this IServiceCollection services)
    {
        services.AddSingleton<StartupGate>();
        services.AddSingleton(TimeProvider.System);
        services.AddHostedService<WarmUpService>();
        // Singleton so the cache survives between probes (AddCheck<T> resolves T from DI if registered).
        services.AddSingleton<AccountStoreCheck>();
        services.AddHealthChecks()
            .AddCheck("self", () => HealthCheckResult.Healthy(), tags: [Live])
            .AddCheck<StartupGateCheck>("startup", tags: [Ready])
            // On the routed endpoint a store failure is Degraded (200); on /health/deps it is Unhealthy.
            .AddCheck<AccountStoreCheck>("account-store", failureStatus: HealthStatus.Degraded, tags: [Ready])
            .AddCheck<AccountStoreCheck>("account-store-deep", failureStatus: HealthStatus.Unhealthy, tags: [Deps]);
        return services;
    }

    public static WebApplication MapHarbourlineHealth(this WebApplication app)
    {
        app.MapHealthChecks("/health/live", Options(Live));
        app.MapHealthChecks("/health/ready", Options(Ready));
        // Internal only: block /health/deps at the ALB listener (or bind it to a separate port).
        app.MapHealthChecks("/health/deps", Options(Deps));
        return app;
    }

    private static HealthCheckOptions Options(string tag) => new()
    {
        Predicate = r => r.Tags.Contains(tag),
        ResponseWriter = WriteMinimalAsync,
        // Defaults: Healthy → 200, Degraded → 200, Unhealthy → 503. Stated explicitly for reviewers.
        ResultStatusCodes =
        {
            [HealthStatus.Healthy] = StatusCodes.Status200OK,
            [HealthStatus.Degraded] = StatusCodes.Status200OK,
            [HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable,
        },
    };

    // Status per check only: no descriptions, exceptions, host names or versions.
    private static Task WriteMinimalAsync(HttpContext context, HealthReport report)
    {
        context.Response.ContentType = "application/json";
        context.Response.Headers.CacheControl = "no-store";
        return context.Response.WriteAsync(JsonSerializer.Serialize(new
        {
            status = report.Status.ToString(),
            checks = report.Entries.ToDictionary(e => e.Key, e => e.Value.Status.ToString()),
        }));
    }
}
Program.cs
using Harbourline.HealthDemo;

// In-image probe for the ECS container health check: chiseled images have no curl or shell,
// so the app itself can act as the probe. Exit code 0 = healthy, 1 = unhealthy.
if (args is ["--probe", var probeUrl])
{
    using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(3) };
    try
    {
        using var response = await http.GetAsync(probeUrl);
        return response.IsSuccessStatusCode ? 0 : 1;
    }
    catch
    {
        return 1;
    }
}

var builder = WebApplication.CreateBuilder(args);

// Graceful shutdown budget: keep it at or below the ECS stopTimeout (30 s default, 120 s max on Fargate).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));

builder.Services.AddSingleton<IAccountStore, SimulatedAccountStore>();
builder.Services.AddHarbourlineHealth();

var app = builder.Build();

app.MapHarbourlineHealth();
app.MapGet("/accounts/{id}", (string id) => Results.Ok(new { accountId = id, currency = "GBP" }));

// Lab helper to simulate a database outage. Development only.
if (app.Environment.IsDevelopment())
{
    app.MapPost("/dev/store/{up:bool}", (bool up, IAccountStore store) =>
    {
        ((SimulatedAccountStore)store).Up = up;
        return Results.NoContent();
    });
}

await app.RunAsync();
return 0;

public sealed class SimulatedAccountStore : IAccountStore
{
    public volatile bool Up = true;
    public Task<bool> PingAsync(CancellationToken cancellationToken) => Task.FromResult(Up);
}

public partial class Program;
tests/Harbourline.HealthDemo.Tests/HealthTests.cs
using System.Net;
using Harbourline.HealthDemo;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;
using Xunit;

public class HealthTests(WebApplicationFactory<Program> factory) : IClassFixture<WebApplicationFactory<Program>>
{
    private async Task<HttpClient> ReadyClientAsync()
    {
        var client = factory.CreateClient();
        factory.Services.GetRequiredService<StartupGate>().MarkReady(); // skip the 3 s warm-up in tests
        await Task.CompletedTask;
        return client;
    }

    private SimulatedAccountStore Store => (SimulatedAccountStore)factory.Services.GetRequiredService<IAccountStore>();

    [Fact]
    public async Task Live_is_200_even_when_the_store_is_down()
    {
        var client = await ReadyClientAsync();
        Store.Up = false;
        try
        {
            var response = await client.GetAsync("/health/live", TestContext.Current.CancellationToken);
            Assert.Equal(HttpStatusCode.OK, response.StatusCode);
        }
        finally { Store.Up = true; }
    }

    [Fact]
    public async Task Ready_reports_degraded_but_200_when_the_shared_store_is_down()
    {
        var client = await ReadyClientAsync();
        Store.Up = false;
        try
        {
            await Task.Delay(TimeSpan.FromSeconds(11), TestContext.Current.CancellationToken); // let any cached result expire
            var response = await client.GetAsync("/health/ready", TestContext.Current.CancellationToken);
            var body = await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken);
            Assert.Equal(HttpStatusCode.OK, response.StatusCode);
            Assert.Contains("Degraded", body);
            Assert.DoesNotContain("Exception", body);
        }
        finally { Store.Up = true; }
    }

    [Fact]
    public async Task Ready_is_503_while_stopping()
    {
        // Own factory: closing the gate is permanent, so it must not affect the shared fixture.
        await using var own = new WebApplicationFactory<Program>();
        var client = own.CreateClient();
        var gate = own.Services.GetRequiredService<StartupGate>();
        gate.MarkReady();
        gate.MarkStopping();
        var response = await client.GetAsync("/health/ready", TestContext.Current.CancellationToken);
        Assert.Equal(HttpStatusCode.ServiceUnavailable, response.StatusCode);
    }
}

The test project follows CONVENTIONS.md: xunit.v3 4.0.3, Microsoft.AspNetCore.Mvc.Testing 10.0.15, <OutputType>Exe</OutputType>, Microsoft.Testing.Platform runner. The 11-second wait in the Degraded test lets the 10-second dependency cache expire; inject a fake TimeProvider (from Microsoft.Extensions.TimeProvider.Testing) to make it instant.

Code explained line by line

  1. StartupGate — two volatile flags read by the readiness check on every probe. Volatile because they are written by one thread (hosted service, shutdown callback) and read by request threads.
  2. StartupGateCheck — the core of readiness: Unhealthy (503) before warm-up finishes and after shutdown begins. These are instance-local states that the ALB should act on.
  3. AccountStoreCheck — the dependency check:
    • caches its result for 10 seconds, so 3 ALB nodes × N tasks × every 15 seconds never become a meaningful load on the database;
    • uses a 2-second timeout so a hanging dependency cannot make the health endpoint exceed the ALB's 5-second timeout;
    • swallows exception details, because health responses are anonymous;
    • returns context.Registration.FailureStatus, so the same class is Degraded on /health/ready and Unhealthy on /health/deps.
  4. WarmUpService — a BackgroundService that opens the gate when warm-up completes, and registers an ApplicationStopping callback that closes it as soon as SIGTERM arrives.
  5. AddCheck(..., tags: [...]) + Predicate — tags select which checks run on which endpoint. Liveness runs only self, which can never fail while the process can serve HTTP.
  6. ResultStatusCodes — the ASP.NET Core defaults (Degraded → 200) written out, so a reviewer sees the decision.
  7. WriteMinimalAsync — status names only, plus Cache-Control: no-store so no proxy caches a health result.
  8. args is ["--probe", var probeUrl] — a list pattern: when the container health check runs dotnet app.dll --probe http://127.0.0.1:8080/health/live, the app makes one HTTP call and exits with 0 or 1, without starting the web host. Cost: each probe starts a short-lived .NET process (tens of MB for a fraction of a second); on a 512 MB task with a 30-second interval this is acceptable, but measure. The alternative is no container health check at all, relying on the ALB.
  9. HostOptions.ShutdownTimeout = 25 s — how long the host waits for in-flight requests and hosted services after SIGTERM. Keep it below ECS stopTimeout (30 s by default) so the process exits cleanly before SIGKILL.
  10. /dev/store/{up} — a Development-only switch to simulate an outage locally and watch the endpoints change.
  11. Tests use WebApplicationFactory<Program> — the real pipeline in memory. They prove the behaviour that matters: liveness ignores dependencies, readiness degrades instead of failing for a shared dependency, and readiness fails during shutdown.

What graceful shutdown looks like on ECS

  1. ECS decides to stop a task (deployment, scale-in, replacement).
  2. ECS deregisters the task from the target group; the ALB stops sending new requests and keeps existing connections for up to the deregistration delay (30 s in our settings).
  3. ECS sends SIGTERM to the container. .NET raises ApplicationStopping; our gate closes; Kestrel stops accepting new connections and waits for in-flight requests up to ShutdownTimeout.
  4. If the process has not exited after stopTimeout, ECS sends SIGKILL.

Keep: request timeout ≤ deregistration delay, and ShutdownTimeout < stopTimeout.

Security considerations

⚠ Learning shortcut — not for production

The demo maps /health/deps on the same port as the public API and includes a Development-only endpoint that toggles the simulated store. In production: serve deep checks on an internal-only listener or block the path with an ALB listener rule (fixed 404 response for /health/deps*), remove all test switches, and publish dependency status as CloudWatch metrics for dashboards instead of an HTTP endpoint.

  • Health endpoints are anonymous by design — treat their output as public. No stack traces, no exception messages, no versions or build numbers that help an attacker pick exploits.
  • Keep them cheap and rate-independent (cached checks, timeouts) so they cannot be used to amplify load on your database.
  • Exclude health probes from request logs at Information level (the lab logs them only at Verbose), otherwise probes dominate log volume and cost.
  • Do not put health endpoints behind authentication middleware by accident: .AllowAnonymous() (as the lab does) when a fallback authorisation policy is configured.
  • A health check that makes outbound calls needs the same least-privilege permissions as the app; the lab's DynamoDB check needs dynamodb:DescribeTable, which the task role grants explicitly.

Banking use case

After the Monday incident described at the start, Harbourline's platform team publishes a health check standard for all ECS services:

  • /health/live: process only. Used by the container health check where an in-image probe exists; otherwise omitted.
  • /health/ready: start-up gate, configuration validated (thanks to ValidateOnStart), shutdown flag; shared dependencies (PostgreSQL, the core-banking adapter, the sanctions service) as Degraded with cached checks.
  • /health/deps: blocked at the ALB, scraped by a small internal job that publishes per-dependency CloudWatch metrics; alarms page the owning team.
  • Target group: 15/5/2/3, matcher 200, deregistration delay matched to the longest request timeout (30 s). Grace period: measured p99 start-up + 50%.

The next primary database failover produced 40 seconds of 503 ProblemDetails responses for writes — as designed — with zero task replacements, and the incident report took 10 minutes to write instead of a day.

Common mistakes

  • Liveness checks the database → every task killed during a database blip → liveness checks the process only.
  • Readiness Unhealthy for a shared dependency on ECS → restart loop → Degraded, alarms, and request-level resilience.
  • Health check path requires authentication → every target unhealthy → AllowAnonymous().
  • Grace period shorter than start-up → tasks killed before they finish starting → measure start-up and add margin.
  • Grace period of 10 minutes "to be safe" → genuinely broken tasks keep failing for 10 minutes → set it from measurements.
  • Default healthy threshold (5 × 30 s) → deployments and scale-out take minutes longer → 2–3 × 15 s.
  • HEALTHCHECK CMD curl ... in a chiseled image → container always unhealthy because curl does not exist → in-image probe or no container check.
  • Returning exception details → internal hostnames leaked → minimal response writer.
  • ShutdownTimeout longer than stopTimeout → SIGKILL mid-request → keep it shorter.

Troubleshooting

SymptomLikely causeFix
Targets unhealthy, reason Target.ResponseCodeMismatchEndpoint returns 401/404/503Check path, authentication and readiness logic; curl from inside the VPC
Targets unhealthy, reason Target.TimeoutSecurity group blocks ALB → task, wrong port, or slow checkAllow the ALB security group on the container port; add a timeout and caching to checks
Tasks replaced every few minutesLiveness/readiness fails on a dependency, or grace period too shortRead service events; separate checks; raise the grace period from measurements
Container UNHEALTHY but ALB says healthyContainer health command fails (missing binary, wrong URL, IPv6 localhost)Use 127.0.0.1, an in-image probe, and check the command with docker run locally
502s during every deploymentApp killed before draining; deregistration delay too short for long requestsReadiness off on ApplicationStopping; delay ≥ request timeout; ShutdownTimeout < stopTimeout
Beanstalk health Red: "ELB health is failing"Beanstalk process health check path wrong or authenticatedSet HealthCheckPath; make it anonymous
All targets unhealthy but traffic still flowsALB fail-open behaviourFix the check; do not rely on fail-open
Health requests dominate logsProbes logged at InformationLower the log level for /health paths

Cost and cleanup

What this lesson costs: the code and tests run locally for free. If you apply the settings to the Fargate or Beanstalk environment from the earlier lessons, the costs are those lessons' costs; health checks themselves are free (ALB health checks do not consume LCUs ⚠ VERIFY). The container health check uses a small amount of task CPU and memory. Label: free.

Cleanup: nothing new is created in AWS by this lesson. If you deployed the Fargate lesson's stack to test these settings, follow its teardown, then confirm:

aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --query 'ResourceTagMappingList[].ResourceARN' --profile learn-sandbox --region us-east-1

Exercises

  • 🟢 Guided: Run the demo locally. Call /health/live, /health/ready and /health/deps during the first 3 seconds and after. Then POST /dev/store/false and call them again after 10 seconds. Write down each status code and explain it.
  • 🟡 Challenge: Compute detection and recovery times for three target-group configurations (defaults; 15/5/2/3; 5/2/2/2) and a 20-second start-up. Which configuration risks false failures during garbage-collection pauses? Then add a fourth test proving that /health/ready returns 503 during the first second after start-up.
  • 🔴 Live problem: Ticket INC-6018: "During the 02:00 database maintenance window last night, the Statements API (ECS, 6 tasks) cycled through 41 task replacements, and the deployment circuit breaker marked this morning's release as failed even though it was fine." Review the attached task definition: container health check CMD-SHELL curl -f http://localhost:8080/health || exit 1, ALB path /health (which runs SELECT 1 on PostgreSQL), grace period 0. Produce a root-cause analysis and corrected settings.

Assignment

Add a HEALTH.md to your portfolio repository: your service's three endpoints and what each checks, the AWS settings (target group, grace period, container check, deregistration delay, stop timeout, shutdown timeout) with the reasoning and a timing table, and the WebApplicationFactory tests from this lesson adapted to your API.

Solutions

Knowledge check

Interview questions

  • "Explain liveness vs readiness on AWS." — Liveness: is the process alive? Used by the ECS container health check; failure means replacement. Readiness: should it get traffic? Used by the ALB; on ECS, failure also leads to replacement, which is why readiness must not fail for shared dependencies.
  • "How would you tune health checks for a .NET API on Fargate?" — Measure start-up; grace period from measurements; ALB 15 s interval, 5 s timeout, 2 healthy, 3 unhealthy; deregistration delay ≥ longest request; ShutdownTimeout below stopTimeout; cheap, cached checks with minimal bodies.
  • "A dependency is down. What should the API do?" — Stay up, report Degraded, fail fast per request with ProblemDetails (circuit breaker), serve cached data where safe, alert humans. Do not let the orchestrator restart everything.
  • "How do you do health checks with a distroless or chiseled image?" — No shell or curl: rely on the ALB check, or add an in-image probe mode (the app itself or a tiny native binary) for the container health check.

Certification practice

Completion checklist

  • I can explain liveness, readiness, start-up and dependency checks and who calls each
  • I ran the demo and saw Degraded keep readiness at 200 while the store was down
  • I can calculate detection and recovery time from target-group settings
  • I know the shutdown sequence on ECS and how to configure it
  • I wrote (or ran) the WebApplicationFactory health tests

Key takeaways

  • Only fail a check if the resulting action would fix the problem. Restarts fix stuck processes, not databases.
  • On ECS behind an ALB, ALB-unhealthy means replaced — so readiness must not fail for shared dependencies; use Degraded and alarms.
  • Tune timings deliberately: grace period from measured start-up, 15 s interval, low healthy threshold, short deregistration delay matched to request timeouts.
  • Close readiness on SIGTERM, keep ShutdownTimeout below stopTimeout, and let the ALB drain.
  • Health endpoints are anonymous and public: minimal bodies, cached checks, no secrets or exception text.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page