Elastic Load Balancing for resilience: cross-zone, health checks and zonal shift
Use load balancer behaviour as a resilience tool: choose cross-zone settings, design health checks that detect real failure without causing outages, drain targets cleanly, and move traffic away from a sick Availability Zone.
Assumes: .NET 10 LTS · ASP.NET Core 10 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Explain what cross-zone load balancing does, how it differs between ALB and NLB, and what it costs in capacity planning.
- Design liveness and readiness health checks for an ASP.NET Core service, and say what each must and must not check.
- Calculate how long it takes the load balancer to notice a failed target, and tune interval and thresholds with a reason.
- Explain fail open, deregistration delay, slow start, least outstanding requests and Automatic Target Weights, and which combinations are invalid.
- Describe zonal shift and zonal autoshift in Amazon Application Recovery Controller, and when they do nothing.
- Add a readiness gate to a .NET service that drains cleanly on shutdown.
Prerequisites
- Elastic Load Balancing: ALB, NLB and Gateway Load Balancer — listeners, target groups and the basic health-check setup. This lesson goes deeper on failure behaviour and does not repeat the basics.
- Health checks in ASP.NET Core —
IHealthCheckand endpoint mapping - Auto Scaling everywhere — capacity per AZ
- Tools: Docker for the lab tests; optional AWS CLI v2 with
learn-sandbox
Why this matters in banking
Two incidents, both caused by load balancer configuration rather than by AWS.
Incident A: the health check that took everyone down. Harbourline's team makes the /health endpoint "thorough": it calls the database and Core Banking. One night Core Banking slows down. Every task's health check starts timing out. All targets become unhealthy at once. The load balancer has no healthy target, so it fails open and sends traffic to all of them anyway, but the orchestrator, seeing failed health checks, starts killing and restarting tasks in a loop. A partial degradation (one slow dependency) became a total outage.
Incident B: the lopsided AZ. A team turns cross-zone balancing off at the target group to save inter-AZ transfer cost. AZ-a holds 2 tasks and AZ-b holds 6, but each AZ's load balancer node gets about half the traffic. The two tasks in AZ-a are saturated while six tasks in AZ-b are idle.
Both are fixable once you know how the load balancer decides, and both are the kind of thing an operational-resilience review asks about.
The idea in plain language
A load balancer does three jobs relevant to resilience:
- Spreads requests across targets (and across AZs).
- Detects failed targets using health checks and stops using them.
- Drains targets that are leaving, so in-flight requests finish.
It cannot fix a problem it cannot see, and it can easily create a problem if its view of health is wrong. Think of it as a bank branch manager directing customers to open counters. A counter flagged "closed" because the cashier went to lunch is correct. A counter flagged closed because the printer is slow, when the other counters share that printer, closes the whole branch.
Analogy: the manager checks each counter every 15 seconds ("are you there? can you serve?"). After two missed checks, the counter's lamp turns off. When a counter is closing for the day, the manager first turns its lamp off (drain), waits for the customer at the window to finish, and only then lets the cashier go home.
Nodes, zones and cross-zone balancing
An Application Load Balancer (ALB) places a load balancer node in each AZ you enable. DNS returns the nodes' addresses, so clients spread across nodes. Each node then chooses a target.
- With cross-zone load balancing on, every node can send requests to targets in every enabled AZ. Traffic is then spread by target count, regardless of which node received it.
- With it off, each node sends only to targets in its own AZ. Traffic is split by node (roughly evenly across AZs), so each AZ must hold a proportionate amount of capacity.
| Load balancer | Cross-zone default | Notes |
|---|---|---|
| ALB | Always on at the load-balancer level; it cannot be turned off there. A target group can override it to off | Inter-AZ data transfer between ALB and targets is not charged for cross-zone traffic on an ALB ⚠ VERIFY current pricing |
| NLB | Off by default; can be turned on | Cross-zone traffic can incur inter-AZ data transfer charges ⚠ VERIFY |
If you turn it off for a target group, remember the consequences documented for ALBs: stickiness is not supported, and an enabled AZ with no targets will answer HTTP requests with errors, so avoid empty subnets and make sure every target group in the load balancer participates in the same AZs. If you cannot guarantee capacity in every AZ, keep cross-zone on.
Why would you ever turn it off? To keep failure domains separate ("a sick AZ must not affect healthy AZs") or to reduce latency or transfer cost. For most banking APIs, keep it on and size capacity with the formula from the previous lesson.
Health checks: liveness versus readiness
A health check is a request the load balancer sends to each target. It counts consecutive failures and successes against thresholds.
The ALB defaults for a target group (for instance or IP targets): check every 30 seconds, 5-second timeout, 2 consecutive failures to mark unhealthy, 5 consecutive successes to mark healthy. The target's default success code is 200.
So the default detection time for a dead target is roughly:
detection = interval x unhealthy threshold = 30 s x 2 = 60 s (plus up to one more interval of unlucky timing).
And the default time to bring a recovered target back is 30 s x 5 = 150 s. In the lab stack we use interval 15 s, thresholds 2 and 2, so detection is about 30 s and recovery about 30 s.
Two different questions deserve two different endpoints:
| Liveness | Readiness | |
|---|---|---|
| Question | "Is the process fundamentally broken?" | "Should I send this task new requests right now?" |
| Used by | Orchestrator restarts (container health command) | Load balancer health check |
| Should check | The process can respond; no deadlock | Not draining; finished starting; essential local state loaded |
| Must NOT check | Any dependency | Slow or flaky shared dependencies (see below) |
| Failure effect | Task is restarted | Task is removed from rotation, not killed |
Why not check dependencies in readiness? If a shared dependency is down, every task is not ready, so the load balancer has zero healthy targets. At that point the ALB fails open: if all targets in the group are unhealthy, it routes to all of them regardless of health. So the dependency-aware check protected nothing; it only added noise and made the failure look total. The right place to react to a sick dependency is inside the call: timeouts, retries, circuit breaker and fallbacks (Resilience patterns in .NET). There are exceptions: a dependency that is per task (a local cache that must be warm, a configuration file that must load) belongs in readiness.
You can also tell the ALB how many healthy targets are enough before it treats the group as unhealthy: target group health requirements (minimum healthy target count or percentage) apply separately to DNS failover (used when the ALB is a Route 53 failover endpoint) and to unhealthy-state routing. Setting these makes the ALB itself signal "this Region is impaired" once capacity falls below your threshold. ⚠ VERIFY the exact behaviour of each threshold in the current ALB documentation before relying on it for failover.
Deregistration delay, slow start and algorithms
- Deregistration delay (default 300 seconds): after a target is deregistered, it goes to
drainingand keeps its in-flight requests until they finish or the delay ends. If the target closes its connections before the delay elapses, clients receive 5xx errors. Short APIs should lower it (30 seconds in the lab) and make sure the app's shutdown time is shorter than it. - Slow start: a newly healthy target receives a linearly increasing share of traffic for a set period instead of a full share immediately. Helpful for JIT-warm .NET services. It cannot be combined with the least-outstanding-requests or weighted-random algorithms.
- Routing algorithm:
round_robin(default),least_outstanding_requests(favours targets with fewer in-flight requests, good when request cost varies or targets are uneven) andweighted_random. - Automatic Target Weights (ATW): the ALB compares each target's error rate (5xx and connection failures) with its peers and flags anomalous targets, even when they still pass health checks. Detection needs at least three healthy targets. Mitigation (automatically sending less traffic to anomalous targets) works only with the weighted-random algorithm.
Zonal shift and zonal autoshift
Zonal shift is a capability of Amazon Application Recovery Controller (ARC). You tell ARC "shift traffic for this resource away from AZ use1-az1". For a supported load balancer, ARC marks that AZ as unavailable so new traffic goes to the other AZs. It is a fast, deliberate action for an AZ-level impairment that your own health checks might not catch (for example, elevated latency from one AZ's network). Key facts:
- Supported resources: ALB and NLB (with cross-zone enabled or disabled), EC2 Auto Scaling groups and EKS clusters.
- Each shift must have an expiry, initially up to three days, and you can cancel or extend it.
- It can take minutes for existing connections and DNS caches to finish moving.
- It does nothing if the load balancer is in fail-open state, because ARC cannot force an AZ to be unhealthy while the ALB is ignoring health.
- If several load balancers share the same targets, a shift on a cross-zone-enabled one removes target capacity for all of them.
- Before you shift, the remaining AZs must have the capacity (that is the static-stability formula again). Shifting traffic onto too few tasks turns an AZ problem into an overload.
Zonal autoshift lets AWS start and end shifts for you when it detects an AZ impairment, for resources you enable. Many teams enable it only after practising manual shifts and running practice runs. ⚠ VERIFY current autoshift behaviour, practice-run requirements and pricing in the ARC documentation before enabling it in a regulated service.
What problem the AWS service solves
Elastic Load Balancing removes single servers and single AZs as points of failure for request traffic, and it does it as a managed, multi-AZ service. For resilience specifically it gives you automatic removal of unhealthy targets, AZ-aware distribution, graceful draining and, with ARC, a deliberate way to avoid an impaired AZ.
Be honest about the limits:
- It is not a circuit breaker. A target that passes health checks but returns slow or erroring responses keeps receiving traffic (ATW helps, but only with the right algorithm and at least three healthy targets).
- Health checks are shallow by design. They sample one path from the load balancer's nodes, not what real users do.
- A load balancer is regional. It does not fail over across Regions. For that, use Route 53 or Global Accelerator (Failover with Route 53).
- Health-check traffic is real traffic and should not hit an expensive endpoint or one behind authentication.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure Application Gateway / Load Balancer health probes | ALB / NLB target group health checks | Thresholds count consecutive results; both are configured per target group |
| Zone-redundant gateway | ALB across enabled AZs, cross-zone on | The ALB is zonal-node-based; you choose which AZs (subnets) to enable |
| Connection draining | Deregistration delay | Default 300 seconds is long for short APIs; tune it |
| Azure Front Door origin health + priority | Route 53 / Global Accelerator for cross-Region | Cross-Region is a separate layer on AWS |
| AKS readiness vs liveness probes | Same ASP.NET Core pattern; ALB uses the readiness path | The ALB check is the readiness check for ECS tasks |
| Azure Traffic Manager manual endpoint disable | ARC zonal shift (AZ) or routing controls (Region) | Zonal shift is AZ-scoped and always has an expiry |
IHealthCheck + MapHealthChecks | Identical in ASP.NET Core | What you put in the check matters more than the API |
Architecture
How a failing target is handled over time:
Security-boundary view: the load balancer security group accepts client traffic and the task security group accepts traffic only from the load balancer's security group, so health checks and requests share one narrow path. Health endpoints must not leak internal detail: return status only, no stack traces or dependency names, and never expose a detailed health page publicly.
Step-by-step implementation
Split liveness and readiness in the service
Map two endpoints: /health/live (always 200 when the process responds, no checks) and /health/ready (a readiness gate that fails while the process is draining). Point the ALB target group at /health/ready.
Add the readiness gate
Register ReadinessGate from the lab library as the single check tagged ready. When SIGTERM arrives, ApplicationStopping fires and the gate flips to unhealthy so the load balancer stops sending new requests while existing ones finish.
Make shutdown shorter than the deregistration delay
Set HostOptions.ShutdownTimeout to about 25 seconds, the ECS stop timeout to 35 seconds, and the target group deregistration delay to 30 seconds. The order is: deregister (stop new traffic), drain, app exits, container stops.
Tune health check timing
Choose interval and thresholds from a detection budget. For the Accounts API we accept about 30 seconds to notice a dead task, so interval 15 s and unhealthy threshold 2. Keep the timeout well below the interval, and keep the check cheap.
Choose the routing algorithm
Use least_outstanding_requests when request cost varies (balance lookups are cheap, statement generation is not). Do not combine it with slow start. If you need anomaly mitigation, use weighted_random instead.
Practise a zonal shift in a non-production account
Deploy the lab stack, start a shift away from one AZ for a few minutes, watch traffic and error rates, then cancel it. Record the time until traffic moved, which feeds your runbook.
AWS Console instructions
- Health checks: Console → EC2 → Load Balancing → Target Groups → your group → Health checks → Edit. Set path, interval, timeout and thresholds. The Targets tab shows each target's health and reason code.
- Attributes: Target Groups → your group → Attributes → Edit: deregistration delay, load balancing algorithm, slow start, cross-zone load balancing, stickiness and target group health requirements.
- Zonal shift: Console → Route 53 Application Recovery Controller → Zonal shift (labels may change; search "zonal shift") ⚠ VERIFY. Choose the resource, the AZ to move away from, an expiry and a comment.
AWS CLI
PROFILE=learn-sandbox
REGION=us-east-1
TG_ARN=$(aws elbv2 describe-target-groups --names hl-m12-account-api --profile $PROFILE --region $REGION \
--query "TargetGroups[0].TargetGroupArn" --output text)
# Health of each target, including the reason when it is not healthy.
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" --profile $PROFILE --region $REGION \
--query "TargetHealthDescriptions[].{Target:Target.Id,AZ:Target.AvailabilityZone,State:TargetHealth.State,Reason:TargetHealth.Reason}" \
--output table
# Include anomaly-detection results (Automatic Target Weights).
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" --include AnomalyDetection --profile $PROFILE --region $REGION
# Tune target group attributes: 30 s draining, least-outstanding-requests routing.
aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" --profile $PROFILE --region $REGION --attributes \
Key=deregistration_delay.timeout_seconds,Value=30 \
Key=load_balancing.algorithm.type,Value=least_outstanding_requests
# Zonal shift (ARC): move traffic away from one AZ for 30 minutes. Use the ALB ARN and a zone ID.
ALB_ARN=$(aws elbv2 describe-load-balancers --names hl-m12-account-api --profile $PROFILE --region $REGION \
--query "LoadBalancers[0].LoadBalancerArn" --output text)
aws arc-zonal-shift start-zonal-shift --resource-identifier "$ALB_ARN" --away-from use1-az1 \
--expires-in 30m --comment "Lab 12: practising a zonal shift" --profile $PROFILE --region $REGION
# See active shifts, and cancel when finished.
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile $PROFILE --region $REGION
aws arc-zonal-shift cancel-zonal-shift --zonal-shift-id <id from the list> --profile $PROFILE --region $REGION⚠ VERIFY the arc-zonal-shift command names, the --expires-in format and whether the ALB must first be registered as a managed resource in your CLI version. The lab's ALB is internet-facing and named hl-m12-account-api in the Terraform stack. If you deployed with CDK, find the generated name with aws elbv2 describe-load-balancers.
Infrastructure as Code
// Excerpt from labs/m12-resilient-account-api/infra/cdk/Harbourline.Resilience.Cdk/AccountApiHaStack.cs
var targetGroup = listener.AddTargets("Api", new AddApplicationTargetsProps
{
Port = props.ContainerPort,
Protocol = ApplicationProtocol.HTTP,
Targets = [service],
DeregistrationDelay = Duration.Seconds(30),
LoadBalancingAlgorithmType = TargetGroupLoadBalancingAlgorithmType.LEAST_OUTSTANDING_REQUESTS,
HealthCheck = new Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck
{
Path = props.HealthCheckPath, // point this at /health/ready for the real API
Interval = Duration.Seconds(15),
Timeout = Duration.Seconds(5),
HealthyThresholdCount = 2,
UnhealthyThresholdCount = 2,
},
});
// Optional: tell the ALB when too few healthy targets means "this Region is impaired" (use a number that fits your fleet).
targetGroup.SetAttribute("target_group_health.dns_failover.minimum_healthy_targets.percentage", "50");
targetGroup.SetAttribute("target_group_health.unhealthy_state_routing.minimum_healthy_targets.percentage", "50");The lab keeps the percentages at off so a three-task demo does not flip into an impaired state by itself. Choose real values with your capacity plan.
Complete code
An ASP.NET Core Program.cs showing the two-endpoint pattern, plus the ReadinessGate from the lab library. The gate is covered by ReadinessGateTests.
using Harbourline.Resilience;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;
var builder = WebApplication.CreateBuilder(args);
// Shutdown must finish BEFORE the target group's deregistration delay (30 s) and the ECS stop timeout (35 s).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));
// One readiness check, tagged "ready". It does NOT call the database or Core Banking.
builder.Services.AddSingleton<ReadinessGate>();
builder.Services.AddHealthChecks()
.AddCheck<ReadinessGate>("draining", tags: ["ready"]);
var app = builder.Build();
// Liveness: the process answers. No checks at all.
app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = _ => false });
// Readiness: only checks tagged "ready". This is the path the ALB health check uses.
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ready"),
ResultStatusCodes =
{
[HealthStatus.Healthy] = StatusCodes.Status200OK,
[HealthStatus.Degraded] = StatusCodes.Status200OK,
[HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable,
},
});
app.MapGet("/accounts/{id}/balance", (string id) => Results.Ok(new { accountId = id, balance = 125.40m, currency = "GBP" }));
app.Run();using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Hosting;
namespace Harbourline.Resilience;
public sealed class ReadinessGate : IHealthCheck
{
private volatile bool _draining;
public ReadinessGate(IHostApplicationLifetime lifetime)
{
// ApplicationStopping fires when SIGTERM arrives (ECS stop, Ctrl+C).
lifetime.ApplicationStopping.Register(BeginDraining);
}
public bool IsDraining => _draining;
public void BeginDraining() => _draining = true;
public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) =>
Task.FromResult(_draining
? HealthCheckResult.Unhealthy("Draining: the process is shutting down.")
: HealthCheckResult.Healthy("Ready."));
}Code explained line by line
Configure<HostOptions>(o => o.ShutdownTimeout = 25 s): the host waits up to 25 seconds for in-flight work when stopping. It must be shorter than the deregistration delay (30 s) and the ECS stop timeout (35 s), otherwise the platform kills the process mid-request and clients see 5xx.AddSingleton<ReadinessGate>()plusAddCheck<ReadinessGate>("draining", tags: ["ready"]): one instance is shared so the gate's state is the same everywhere; thereadytag lets the readiness endpoint select it.MapHealthChecks("/health/live", Predicate = _ => false): runs no checks, so it returns healthy whenever the process can answer. An orchestrator can use it to restart a hung process.Predicate = check => check.Tags.Contains("ready"): the readiness endpoint runs only tagged checks.ResultStatusCodes: maps unhealthy to503so the ALB (which only looks at status codes) sees failure.Degradedstays200, since a degraded-but-serving task should keep receiving traffic.- In
ReadinessGate,lifetime.ApplicationStopping.Register(BeginDraining)wires SIGTERM to the flag.volatilemakes the write visible to other threads immediately. - The gate never calls a downstream service: that is the point of the design.
Security considerations
- Health endpoints are unauthenticated by necessity (the load balancer cannot log in). Keep them minimal: a status code and a short word. Restrict who can reach them with security groups (ALB to task only), and do not map detailed health UIs on the public listener.
- Do not expose dependency names or errors in health responses; they help attackers map your estate.
- Least privilege for zonal shift: only a small operations role should hold
arc-zonal-shift:StartZonalShift,UpdateZonalShiftandCancelZonalShift. Shifting traffic is a high-impact action; log it in CloudTrail and require a ticket reference in the comment. - TLS: health checks use HTTP inside the VPC in the lab. Production listeners terminate TLS; use HTTPS between ALB and tasks if policy requires encryption in transit end to end. Route 53 HTTPS health checks do not validate certificates, so an expired certificate is not detected by them.
⚠ Learning shortcut — not for production
The lab ALB has an HTTP-only listener open to the internet, the tasks run with public IPs in public subnets, and the placeholder container answers 200 on /. Production uses HTTPS with an ACM certificate and a modern TLS policy, WAF in front, private subnets for tasks, and a real /health/ready path. Never point a health check at a page that needs a session cookie or a token.
Banking use case
Harbourline's Accounts API runs 9 tasks across 3 AZs behind an ALB. The standards the platform team adopts:
- Every service exposes
/health/liveand/health/ready; the ALB uses/health/readyonly; readiness never calls a shared dependency. - Target group: interval 15 s, unhealthy threshold 2, healthy threshold 2, timeout 5 s; deregistration delay 30 s; app shutdown 25 s; ECS stop timeout 35 s.
least_outstanding_requestsfor APIs with mixed request cost.- Cross-zone stays on; capacity per AZ follows the AZ-loss formula.
- A quarterly "zonal shift drill" in a pre-production account, timed and recorded for the operational-resilience evidence pack.
All values here are fictional teaching defaults; your own load tests and risk appetite set the real ones.
Common mistakes
- Dependency checks in the ALB health check. One slow downstream removes every task, and the ALB fails open anyway.
- Health path behind authentication or redirects. Returns 301 or 401, so every target is "unhealthy". Use an anonymous path and a matching success code.
- Timeout close to the interval. Slow-but-alive tasks flap in and out of service.
- Default deregistration delay (300 s) on a short API. Deployments and scale-in take much longer than needed. The opposite error, an app that exits immediately on SIGTERM, causes 502 and 5xx during deployments.
- Cross-zone off with uneven capacity. One AZ runs hot while others are idle.
- Empty subnet enabled on the load balancer. Requests to that AZ's node fail.
- Assuming zonal shift works in fail-open. It has no effect when all targets are unhealthy.
- Relying on health checks for gray failures. Slow and erroring targets still pass; add breakers, timeouts, ATW and alarms on latency and error rate.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
All targets unhealthy with reason Target.ResponseCodeMismatch | Health path returns something other than the success code (redirect, 401) | Use an anonymous path; set the matcher; test with curl -i from inside the VPC |
Targets unhealthy with Target.Timeout | App slow to start or check too heavy; timeout too short | Raise the grace period; make the check cheap; align timeout and interval |
| Targets flap between healthy and unhealthy | Check hits a dependency or the thresholds are too tight | Make readiness local-only; raise thresholds |
| 5xx during every deployment | App exits before the load balancer finishes draining | Add the readiness gate; shutdown shorter than the deregistration delay; handle SIGTERM |
| One AZ's tasks are saturated, others idle | Cross-zone off or uneven task distribution | Turn cross-zone on at the target group; enable service AZ rebalancing |
| Zonal shift started but traffic still reaches the AZ | Load balancer in fail-open, or existing connections and DNS caches | Check target health; wait for connections to drain; confirm the shift is ACTIVE |
describe-target-health shows unused | AZ not enabled for the load balancer or target group not used by a listener | Enable the AZ; attach the target group to a listener |
Cost and cleanup
Reading and local code are free. Deploying the lab stack costs about USD 0.10 per hour (ALB, three Fargate tasks) ⚠ VERIFY current prices; zonal shift itself has no per-shift charge that we know of ⚠ VERIFY, and zonal autoshift practice runs are best checked against the current ARC pricing page. Destroy the stack within the hour.
# CDK
cd labs/m12-resilient-account-api/infra/cdk && cdk destroy -c owner=<your-name> --profile learn-sandbox
# or Terraform
cd labs/m12-resilient-account-api/infra/terraform && terraform destroy
# Cancel any zonal shift left active (they expire, but do not leave them).
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile learn-sandbox --region us-east-1
# Confirm nothing tagged for the lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
--profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"Exercises
- 🟢 Guided: Run
ReadinessGateTests. Then add/health/liveand/health/readyto a small ASP.NET Core app, start it, and call both. Send Ctrl+C and watch/health/readyflip to 503 before the process exits. - 🟡 Challenge: Your check interval is 30 s with an unhealthy threshold of 3 and a healthy threshold of 5. Calculate worst-case detection and recovery times, then propose settings for a 30-second detection budget and say what you give up.
- 🔴 Live problem: "During a Core Banking slowdown, the Accounts API became completely unavailable even though 90% of requests do not need Core Banking." You find the ALB health check calls
/healthwhich tests Core Banking and the database. Write the root cause, the fix, and the monitoring you will add so a slow dependency is visible without removing tasks.
Assignment
Create docs/health-check-standard.md for Harbourline: required endpoints, what readiness may and may not check, timing defaults and their reasoning, deregistration and shutdown rules, and a decision table for cross-zone settings. Include a link to your test results.
Solutions
After Ctrl+C, ApplicationStopping fires and ReadinessGate.BeginDraining sets the flag, so /health/ready returns 503 while /health/live still returns 200 until the process exits. That ordering is exactly what a load balancer needs: stop sending new requests, then let in-flight ones finish within the shutdown timeout.
Worst-case detection: 30 s x 3 = 90 s, plus up to one extra interval of timing, so up to 120 s. Recovery: 30 s x 5 = 150 s. For a 30-second detection budget use interval 10 s and unhealthy threshold 3 (30 s), or interval 15 s with threshold 2 (30 s). The trade-off: faster checks mean more health-check traffic and a higher chance of marking a briefly slow task unhealthy (flapping), so keep the timeout well below the interval and the check cheap. ALB health-check minimum interval and timeout limits apply ⚠ VERIFY.
Root cause: the ALB health check had a hard dependency on Core Banking, so when it slowed, every task failed the check at once; with no healthy targets the ALB failed open and orchestration restarted tasks, turning a partial degradation into a full outage. Fix: ALB points at a local-only readiness endpoint (/health/ready, draining only); liveness checks nothing. Protect the dependency calls with timeouts, a circuit breaker and fallbacks (cached last-known balance). Monitoring: a dependency-latency and error-rate metric per downstream, breaker state metric, and alarms on those, so humans are told about the slow dependency without it changing routing.
Knowledge check
Answer: If the dependency fails, every target fails the check simultaneously, leaving no healthy targets. The ALB then fails open (sends traffic to all targets anyway), so the check protected nothing and may trigger restart loops. Handle dependency failure inside calls with timeouts, breakers and fallbacks.
Answer: It is always on at the load-balancer level and cannot be turned off there; a target group can override it to off. With it on, every node can send to targets in any enabled AZ; with it off, each node sends only to targets in its own AZ, so each AZ must hold proportional capacity.
Answer: About 60 seconds (interval times unhealthy threshold), plus up to one more interval depending on timing. Reduce the interval or threshold for faster detection, at the cost of more checks and more flapping risk.
Answer: The load balancer is in fail-open state (all targets unhealthy) so ARC cannot force the AZ unhealthy; or existing connections and DNS caches are still draining and it needs a few minutes. Also check the resource is managed in ARC and the shift is active.
Answer: Least outstanding requests and weighted random cannot be used with slow start mode. Round robin can.
Interview questions
- "How do you do zero-downtime deployments behind an ALB?" Readiness that flips to unhealthy on SIGTERM, shutdown shorter than the deregistration delay, rolling or blue/green updates with a minimum-healthy percentage, and a deployment circuit breaker that rolls back.
- "What goes in a health check?" Liveness checks nothing external. Readiness checks only what makes this task unable to serve right now (draining, still starting, local state). Dependency health belongs in metrics and breakers.
- "What is fail open?" If no target in a group is healthy, the ALB sends traffic to all of them rather than none, betting that some can still answer. It is why health checks alone cannot protect against a shared failure.
- "When would you use zonal shift?" When one AZ is impaired in a way health checks do not catch, and the other AZs have the capacity to absorb the traffic. Always set an expiry, and know it will not help in fail-open.
Certification practice
An application behind an Application Load Balancer must stop receiving traffic on any instance whose web process hangs, but instances should not be restarted or replaced simply because a shared database is slow. Which health check design meets this?
A. An HTTP health check on a path that queries the database. B. An HTTP health check on a lightweight path that verifies the process responds. C. A TCP check on the database port. D. No health check.
Answer: B. A lightweight local check detects a hung process without coupling instance health to a shared dependency. A would mark every instance unhealthy when the database slows. C checks the wrong component. D leaves failed instances in rotation.
After each deployment, users see HTTP 502 and 504 errors for about 20 seconds. The application exits as soon as it receives SIGTERM. The target group deregistration delay is 300 seconds. What should the team change?
A. Increase the instance size. B. Make the application stop accepting new work on SIGTERM but finish in-flight requests before exiting, with shutdown time shorter than the deregistration delay. C. Disable health checks. D. Use a Network Load Balancer.
Answer: B. If the target closes connections before the deregistration delay elapses, clients get 5xx errors. Draining gracefully fixes it. A, C and D do not address premature exits.
An Availability Zone has a network problem causing elevated latency, while instances still pass load balancer health checks. The remaining AZs have spare capacity. Which action quickly reduces customer impact with minimal changes?
A. Terminate all instances in the affected AZ. B. Start a zonal shift in Application Recovery Controller away from the affected AZ for a limited time. C. Fail over to another Region. D. Disable cross-zone load balancing permanently.
Answer: B. Zonal shift moves traffic away from an AZ without waiting for health checks to fail, and is time-limited. A is destructive. C is far larger than needed. D does not remove the impaired AZ and has side effects.
Completion checklist
- I can explain cross-zone balancing for ALB and NLB and when to turn it off
- I can write a liveness and a readiness endpoint and explain why readiness has no dependency checks
- I ran
ReadinessGateTestsand the drain demo - I calculated detection and recovery times for my own health-check settings
- I ran the cleanup steps and confirmed nothing is still billing
Key takeaways
- The ALB fails open when all targets are unhealthy, so dependency-aware health checks add risk, not safety.
- Detection time is interval times unhealthy threshold; keep checks cheap and local.
- Drain on SIGTERM and keep shutdown shorter than the deregistration delay.
- Zonal shift is a deliberate, time-limited AZ evacuation that needs spare capacity and does nothing in fail-open.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Auto Scaling everywhere: EC2, ECS, Lambda, DynamoDB and Aurora
Configure scaling for compute and data layers, understand what each one scales on, and size minimum capacity so that losing an Availability Zone does not need a scale-out to survive.
Multi-AZ by default: RDS, Aurora, ElastiCache, ECS and your own services
Know exactly what Multi-AZ means for each service Harbourline uses, which parts AWS does for you and which parts are your job, and make your own stateless .NET services AZ-aware and failover-tolerant.