Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Elastic Load Balancing for resilience: cross-zone, health checks and zonal shift

Use load balancer behaviour as a resilience tool: choose cross-zone settings, design health checks that detect real failure without causing outages, drain targets cleanly, and move traffic away from a sick Availability Zone.

intermediate⏱ 1 h 20 min🟢 FreeSAA-C03SOA-C03

Assumes: .NET 10 LTS · ASP.NET Core 10 · AWS CLI v2 · CDK v2 (Amazon.CDK.Lib 2.277.0) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what cross-zone load balancing does, how it differs between ALB and NLB, and what it costs in capacity planning.
  • Design liveness and readiness health checks for an ASP.NET Core service, and say what each must and must not check.
  • Calculate how long it takes the load balancer to notice a failed target, and tune interval and thresholds with a reason.
  • Explain fail open, deregistration delay, slow start, least outstanding requests and Automatic Target Weights, and which combinations are invalid.
  • Describe zonal shift and zonal autoshift in Amazon Application Recovery Controller, and when they do nothing.
  • Add a readiness gate to a .NET service that drains cleanly on shutdown.

Prerequisites

Why this matters in banking

Two incidents, both caused by load balancer configuration rather than by AWS.

Incident A: the health check that took everyone down. Harbourline's team makes the /health endpoint "thorough": it calls the database and Core Banking. One night Core Banking slows down. Every task's health check starts timing out. All targets become unhealthy at once. The load balancer has no healthy target, so it fails open and sends traffic to all of them anyway, but the orchestrator, seeing failed health checks, starts killing and restarting tasks in a loop. A partial degradation (one slow dependency) became a total outage.

Incident B: the lopsided AZ. A team turns cross-zone balancing off at the target group to save inter-AZ transfer cost. AZ-a holds 2 tasks and AZ-b holds 6, but each AZ's load balancer node gets about half the traffic. The two tasks in AZ-a are saturated while six tasks in AZ-b are idle.

Both are fixable once you know how the load balancer decides, and both are the kind of thing an operational-resilience review asks about.

The idea in plain language

A load balancer does three jobs relevant to resilience:

  1. Spreads requests across targets (and across AZs).
  2. Detects failed targets using health checks and stops using them.
  3. Drains targets that are leaving, so in-flight requests finish.

It cannot fix a problem it cannot see, and it can easily create a problem if its view of health is wrong. Think of it as a bank branch manager directing customers to open counters. A counter flagged "closed" because the cashier went to lunch is correct. A counter flagged closed because the printer is slow, when the other counters share that printer, closes the whole branch.

Analogy: the manager checks each counter every 15 seconds ("are you there? can you serve?"). After two missed checks, the counter's lamp turns off. When a counter is closing for the day, the manager first turns its lamp off (drain), waits for the customer at the window to finish, and only then lets the cashier go home.

Nodes, zones and cross-zone balancing

An Application Load Balancer (ALB) places a load balancer node in each AZ you enable. DNS returns the nodes' addresses, so clients spread across nodes. Each node then chooses a target.

  • With cross-zone load balancing on, every node can send requests to targets in every enabled AZ. Traffic is then spread by target count, regardless of which node received it.
  • With it off, each node sends only to targets in its own AZ. Traffic is split by node (roughly evenly across AZs), so each AZ must hold a proportionate amount of capacity.
Load balancerCross-zone defaultNotes
ALBAlways on at the load-balancer level; it cannot be turned off there. A target group can override it to offInter-AZ data transfer between ALB and targets is not charged for cross-zone traffic on an ALB ⚠ VERIFY current pricing
NLBOff by default; can be turned onCross-zone traffic can incur inter-AZ data transfer charges ⚠ VERIFY

If you turn it off for a target group, remember the consequences documented for ALBs: stickiness is not supported, and an enabled AZ with no targets will answer HTTP requests with errors, so avoid empty subnets and make sure every target group in the load balancer participates in the same AZs. If you cannot guarantee capacity in every AZ, keep cross-zone on.

Why would you ever turn it off? To keep failure domains separate ("a sick AZ must not affect healthy AZs") or to reduce latency or transfer cost. For most banking APIs, keep it on and size capacity with the formula from the previous lesson.

Health checks: liveness versus readiness

A health check is a request the load balancer sends to each target. It counts consecutive failures and successes against thresholds.

The ALB defaults for a target group (for instance or IP targets): check every 30 seconds, 5-second timeout, 2 consecutive failures to mark unhealthy, 5 consecutive successes to mark healthy. The target's default success code is 200.

So the default detection time for a dead target is roughly:

detection = interval x unhealthy threshold = 30 s x 2 = 60 s (plus up to one more interval of unlucky timing).

And the default time to bring a recovered target back is 30 s x 5 = 150 s. In the lab stack we use interval 15 s, thresholds 2 and 2, so detection is about 30 s and recovery about 30 s.

Two different questions deserve two different endpoints:

LivenessReadiness
Question"Is the process fundamentally broken?""Should I send this task new requests right now?"
Used byOrchestrator restarts (container health command)Load balancer health check
Should checkThe process can respond; no deadlockNot draining; finished starting; essential local state loaded
Must NOT checkAny dependencySlow or flaky shared dependencies (see below)
Failure effectTask is restartedTask is removed from rotation, not killed

Why not check dependencies in readiness? If a shared dependency is down, every task is not ready, so the load balancer has zero healthy targets. At that point the ALB fails open: if all targets in the group are unhealthy, it routes to all of them regardless of health. So the dependency-aware check protected nothing; it only added noise and made the failure look total. The right place to react to a sick dependency is inside the call: timeouts, retries, circuit breaker and fallbacks (Resilience patterns in .NET). There are exceptions: a dependency that is per task (a local cache that must be warm, a configuration file that must load) belongs in readiness.

You can also tell the ALB how many healthy targets are enough before it treats the group as unhealthy: target group health requirements (minimum healthy target count or percentage) apply separately to DNS failover (used when the ALB is a Route 53 failover endpoint) and to unhealthy-state routing. Setting these makes the ALB itself signal "this Region is impaired" once capacity falls below your threshold. ⚠ VERIFY the exact behaviour of each threshold in the current ALB documentation before relying on it for failover.

Deregistration delay, slow start and algorithms

  • Deregistration delay (default 300 seconds): after a target is deregistered, it goes to draining and keeps its in-flight requests until they finish or the delay ends. If the target closes its connections before the delay elapses, clients receive 5xx errors. Short APIs should lower it (30 seconds in the lab) and make sure the app's shutdown time is shorter than it.
  • Slow start: a newly healthy target receives a linearly increasing share of traffic for a set period instead of a full share immediately. Helpful for JIT-warm .NET services. It cannot be combined with the least-outstanding-requests or weighted-random algorithms.
  • Routing algorithm: round_robin (default), least_outstanding_requests (favours targets with fewer in-flight requests, good when request cost varies or targets are uneven) and weighted_random.
  • Automatic Target Weights (ATW): the ALB compares each target's error rate (5xx and connection failures) with its peers and flags anomalous targets, even when they still pass health checks. Detection needs at least three healthy targets. Mitigation (automatically sending less traffic to anomalous targets) works only with the weighted-random algorithm.

Zonal shift and zonal autoshift

Zonal shift is a capability of Amazon Application Recovery Controller (ARC). You tell ARC "shift traffic for this resource away from AZ use1-az1". For a supported load balancer, ARC marks that AZ as unavailable so new traffic goes to the other AZs. It is a fast, deliberate action for an AZ-level impairment that your own health checks might not catch (for example, elevated latency from one AZ's network). Key facts:

  • Supported resources: ALB and NLB (with cross-zone enabled or disabled), EC2 Auto Scaling groups and EKS clusters.
  • Each shift must have an expiry, initially up to three days, and you can cancel or extend it.
  • It can take minutes for existing connections and DNS caches to finish moving.
  • It does nothing if the load balancer is in fail-open state, because ARC cannot force an AZ to be unhealthy while the ALB is ignoring health.
  • If several load balancers share the same targets, a shift on a cross-zone-enabled one removes target capacity for all of them.
  • Before you shift, the remaining AZs must have the capacity (that is the static-stability formula again). Shifting traffic onto too few tasks turns an AZ problem into an overload.

Zonal autoshift lets AWS start and end shifts for you when it detects an AZ impairment, for resources you enable. Many teams enable it only after practising manual shifts and running practice runs. ⚠ VERIFY current autoshift behaviour, practice-run requirements and pricing in the ARC documentation before enabling it in a regulated service.

What problem the AWS service solves

Elastic Load Balancing removes single servers and single AZs as points of failure for request traffic, and it does it as a managed, multi-AZ service. For resilience specifically it gives you automatic removal of unhealthy targets, AZ-aware distribution, graceful draining and, with ARC, a deliberate way to avoid an impaired AZ.

Be honest about the limits:

  • It is not a circuit breaker. A target that passes health checks but returns slow or erroring responses keeps receiving traffic (ATW helps, but only with the right algorithm and at least three healthy targets).
  • Health checks are shallow by design. They sample one path from the load balancer's nodes, not what real users do.
  • A load balancer is regional. It does not fail over across Regions. For that, use Route 53 or Global Accelerator (Failover with Route 53).
  • Health-check traffic is real traffic and should not hit an expensive endpoint or one behind authentication.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure Application Gateway / Load Balancer health probesALB / NLB target group health checksThresholds count consecutive results; both are configured per target group
Zone-redundant gatewayALB across enabled AZs, cross-zone onThe ALB is zonal-node-based; you choose which AZs (subnets) to enable
Connection drainingDeregistration delayDefault 300 seconds is long for short APIs; tune it
Azure Front Door origin health + priorityRoute 53 / Global Accelerator for cross-RegionCross-Region is a separate layer on AWS
AKS readiness vs liveness probesSame ASP.NET Core pattern; ALB uses the readiness pathThe ALB check is the readiness check for ECS tasks
Azure Traffic Manager manual endpoint disableARC zonal shift (AZ) or routing controls (Region)Zonal shift is AZ-scoped and always has an expiry
IHealthCheck + MapHealthChecksIdentical in ASP.NET CoreWhat you put in the check matters more than the API

Architecture

How a failing target is handled over time:

Security-boundary view: the load balancer security group accepts client traffic and the task security group accepts traffic only from the load balancer's security group, so health checks and requests share one narrow path. Health endpoints must not leak internal detail: return status only, no stack traces or dependency names, and never expose a detailed health page publicly.

Step-by-step implementation

Split liveness and readiness in the service

Map two endpoints: /health/live (always 200 when the process responds, no checks) and /health/ready (a readiness gate that fails while the process is draining). Point the ALB target group at /health/ready.

Add the readiness gate

Register ReadinessGate from the lab library as the single check tagged ready. When SIGTERM arrives, ApplicationStopping fires and the gate flips to unhealthy so the load balancer stops sending new requests while existing ones finish.

Make shutdown shorter than the deregistration delay

Set HostOptions.ShutdownTimeout to about 25 seconds, the ECS stop timeout to 35 seconds, and the target group deregistration delay to 30 seconds. The order is: deregister (stop new traffic), drain, app exits, container stops.

Tune health check timing

Choose interval and thresholds from a detection budget. For the Accounts API we accept about 30 seconds to notice a dead task, so interval 15 s and unhealthy threshold 2. Keep the timeout well below the interval, and keep the check cheap.

Choose the routing algorithm

Use least_outstanding_requests when request cost varies (balance lookups are cheap, statement generation is not). Do not combine it with slow start. If you need anomaly mitigation, use weighted_random instead.

Practise a zonal shift in a non-production account

Deploy the lab stack, start a shift away from one AZ for a few minutes, watch traffic and error rates, then cancel it. Record the time until traffic moved, which feeds your runbook.

AWS Console instructions

  • Health checks: Console → EC2 → Load Balancing → Target Groups → your group → Health checks → Edit. Set path, interval, timeout and thresholds. The Targets tab shows each target's health and reason code.
  • Attributes: Target Groups → your group → Attributes → Edit: deregistration delay, load balancing algorithm, slow start, cross-zone load balancing, stickiness and target group health requirements.
  • Zonal shift: Console → Route 53 Application Recovery Controller → Zonal shift (labels may change; search "zonal shift") ⚠ VERIFY. Choose the resource, the AZ to move away from, an expiry and a comment.

AWS CLI

PROFILE=learn-sandbox
REGION=us-east-1
TG_ARN=$(aws elbv2 describe-target-groups --names hl-m12-account-api --profile $PROFILE --region $REGION \
  --query "TargetGroups[0].TargetGroupArn" --output text)

# Health of each target, including the reason when it is not healthy.
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" --profile $PROFILE --region $REGION \
  --query "TargetHealthDescriptions[].{Target:Target.Id,AZ:Target.AvailabilityZone,State:TargetHealth.State,Reason:TargetHealth.Reason}" \
  --output table

# Include anomaly-detection results (Automatic Target Weights).
aws elbv2 describe-target-health --target-group-arn "$TG_ARN" --include AnomalyDetection --profile $PROFILE --region $REGION

# Tune target group attributes: 30 s draining, least-outstanding-requests routing.
aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" --profile $PROFILE --region $REGION --attributes \
  Key=deregistration_delay.timeout_seconds,Value=30 \
  Key=load_balancing.algorithm.type,Value=least_outstanding_requests

# Zonal shift (ARC): move traffic away from one AZ for 30 minutes. Use the ALB ARN and a zone ID.
ALB_ARN=$(aws elbv2 describe-load-balancers --names hl-m12-account-api --profile $PROFILE --region $REGION \
  --query "LoadBalancers[0].LoadBalancerArn" --output text)

aws arc-zonal-shift start-zonal-shift --resource-identifier "$ALB_ARN" --away-from use1-az1 \
  --expires-in 30m --comment "Lab 12: practising a zonal shift" --profile $PROFILE --region $REGION

# See active shifts, and cancel when finished.
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile $PROFILE --region $REGION
aws arc-zonal-shift cancel-zonal-shift --zonal-shift-id <id from the list> --profile $PROFILE --region $REGION

⚠ VERIFY the arc-zonal-shift command names, the --expires-in format and whether the ALB must first be registered as a managed resource in your CLI version. The lab's ALB is internet-facing and named hl-m12-account-api in the Terraform stack. If you deployed with CDK, find the generated name with aws elbv2 describe-load-balancers.

Infrastructure as Code

// Excerpt from labs/m12-resilient-account-api/infra/cdk/Harbourline.Resilience.Cdk/AccountApiHaStack.cs
var targetGroup = listener.AddTargets("Api", new AddApplicationTargetsProps
{
    Port = props.ContainerPort,
    Protocol = ApplicationProtocol.HTTP,
    Targets = [service],
    DeregistrationDelay = Duration.Seconds(30),
    LoadBalancingAlgorithmType = TargetGroupLoadBalancingAlgorithmType.LEAST_OUTSTANDING_REQUESTS,
    HealthCheck = new Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck
    {
        Path = props.HealthCheckPath,        // point this at /health/ready for the real API
        Interval = Duration.Seconds(15),
        Timeout = Duration.Seconds(5),
        HealthyThresholdCount = 2,
        UnhealthyThresholdCount = 2,
    },
});

// Optional: tell the ALB when too few healthy targets means "this Region is impaired" (use a number that fits your fleet).
targetGroup.SetAttribute("target_group_health.dns_failover.minimum_healthy_targets.percentage", "50");
targetGroup.SetAttribute("target_group_health.unhealthy_state_routing.minimum_healthy_targets.percentage", "50");

The lab keeps the percentages at off so a three-task demo does not flip into an impaired state by itself. Choose real values with your capacity plan.

Complete code

An ASP.NET Core Program.cs showing the two-endpoint pattern, plus the ReadinessGate from the lab library. The gate is covered by ReadinessGateTests.

Program.cs (Accounts API excerpt)
using Harbourline.Resilience;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);

// Shutdown must finish BEFORE the target group's deregistration delay (30 s) and the ECS stop timeout (35 s).
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));

// One readiness check, tagged "ready". It does NOT call the database or Core Banking.
builder.Services.AddSingleton<ReadinessGate>();
builder.Services.AddHealthChecks()
    .AddCheck<ReadinessGate>("draining", tags: ["ready"]);

var app = builder.Build();

// Liveness: the process answers. No checks at all.
app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = _ => false });

// Readiness: only checks tagged "ready". This is the path the ALB health check uses.
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResultStatusCodes =
    {
        [HealthStatus.Healthy] = StatusCodes.Status200OK,
        [HealthStatus.Degraded] = StatusCodes.Status200OK,
        [HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable,
    },
});

app.MapGet("/accounts/{id}/balance", (string id) => Results.Ok(new { accountId = id, balance = 125.40m, currency = "GBP" }));

app.Run();
src/Harbourline.Resilience/ReadinessGate.cs
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Hosting;

namespace Harbourline.Resilience;

public sealed class ReadinessGate : IHealthCheck
{
    private volatile bool _draining;

    public ReadinessGate(IHostApplicationLifetime lifetime)
    {
        // ApplicationStopping fires when SIGTERM arrives (ECS stop, Ctrl+C).
        lifetime.ApplicationStopping.Register(BeginDraining);
    }

    public bool IsDraining => _draining;

    public void BeginDraining() => _draining = true;

    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) =>
        Task.FromResult(_draining
            ? HealthCheckResult.Unhealthy("Draining: the process is shutting down.")
            : HealthCheckResult.Healthy("Ready."));
}

Code explained line by line

  1. Configure<HostOptions>(o => o.ShutdownTimeout = 25 s): the host waits up to 25 seconds for in-flight work when stopping. It must be shorter than the deregistration delay (30 s) and the ECS stop timeout (35 s), otherwise the platform kills the process mid-request and clients see 5xx.
  2. AddSingleton<ReadinessGate>() plus AddCheck<ReadinessGate>("draining", tags: ["ready"]): one instance is shared so the gate's state is the same everywhere; the ready tag lets the readiness endpoint select it.
  3. MapHealthChecks("/health/live", Predicate = _ => false): runs no checks, so it returns healthy whenever the process can answer. An orchestrator can use it to restart a hung process.
  4. Predicate = check => check.Tags.Contains("ready"): the readiness endpoint runs only tagged checks.
  5. ResultStatusCodes: maps unhealthy to 503 so the ALB (which only looks at status codes) sees failure. Degraded stays 200, since a degraded-but-serving task should keep receiving traffic.
  6. In ReadinessGate, lifetime.ApplicationStopping.Register(BeginDraining) wires SIGTERM to the flag. volatile makes the write visible to other threads immediately.
  7. The gate never calls a downstream service: that is the point of the design.

Security considerations

  • Health endpoints are unauthenticated by necessity (the load balancer cannot log in). Keep them minimal: a status code and a short word. Restrict who can reach them with security groups (ALB to task only), and do not map detailed health UIs on the public listener.
  • Do not expose dependency names or errors in health responses; they help attackers map your estate.
  • Least privilege for zonal shift: only a small operations role should hold arc-zonal-shift:StartZonalShift, UpdateZonalShift and CancelZonalShift. Shifting traffic is a high-impact action; log it in CloudTrail and require a ticket reference in the comment.
  • TLS: health checks use HTTP inside the VPC in the lab. Production listeners terminate TLS; use HTTPS between ALB and tasks if policy requires encryption in transit end to end. Route 53 HTTPS health checks do not validate certificates, so an expired certificate is not detected by them.

⚠ Learning shortcut — not for production

The lab ALB has an HTTP-only listener open to the internet, the tasks run with public IPs in public subnets, and the placeholder container answers 200 on /. Production uses HTTPS with an ACM certificate and a modern TLS policy, WAF in front, private subnets for tasks, and a real /health/ready path. Never point a health check at a page that needs a session cookie or a token.

Banking use case

Harbourline's Accounts API runs 9 tasks across 3 AZs behind an ALB. The standards the platform team adopts:

  • Every service exposes /health/live and /health/ready; the ALB uses /health/ready only; readiness never calls a shared dependency.
  • Target group: interval 15 s, unhealthy threshold 2, healthy threshold 2, timeout 5 s; deregistration delay 30 s; app shutdown 25 s; ECS stop timeout 35 s.
  • least_outstanding_requests for APIs with mixed request cost.
  • Cross-zone stays on; capacity per AZ follows the AZ-loss formula.
  • A quarterly "zonal shift drill" in a pre-production account, timed and recorded for the operational-resilience evidence pack.

All values here are fictional teaching defaults; your own load tests and risk appetite set the real ones.

Common mistakes

  • Dependency checks in the ALB health check. One slow downstream removes every task, and the ALB fails open anyway.
  • Health path behind authentication or redirects. Returns 301 or 401, so every target is "unhealthy". Use an anonymous path and a matching success code.
  • Timeout close to the interval. Slow-but-alive tasks flap in and out of service.
  • Default deregistration delay (300 s) on a short API. Deployments and scale-in take much longer than needed. The opposite error, an app that exits immediately on SIGTERM, causes 502 and 5xx during deployments.
  • Cross-zone off with uneven capacity. One AZ runs hot while others are idle.
  • Empty subnet enabled on the load balancer. Requests to that AZ's node fail.
  • Assuming zonal shift works in fail-open. It has no effect when all targets are unhealthy.
  • Relying on health checks for gray failures. Slow and erroring targets still pass; add breakers, timeouts, ATW and alarms on latency and error rate.

Troubleshooting

SymptomLikely causeFix
All targets unhealthy with reason Target.ResponseCodeMismatchHealth path returns something other than the success code (redirect, 401)Use an anonymous path; set the matcher; test with curl -i from inside the VPC
Targets unhealthy with Target.TimeoutApp slow to start or check too heavy; timeout too shortRaise the grace period; make the check cheap; align timeout and interval
Targets flap between healthy and unhealthyCheck hits a dependency or the thresholds are too tightMake readiness local-only; raise thresholds
5xx during every deploymentApp exits before the load balancer finishes drainingAdd the readiness gate; shutdown shorter than the deregistration delay; handle SIGTERM
One AZ's tasks are saturated, others idleCross-zone off or uneven task distributionTurn cross-zone on at the target group; enable service AZ rebalancing
Zonal shift started but traffic still reaches the AZLoad balancer in fail-open, or existing connections and DNS cachesCheck target health; wait for connections to drain; confirm the shift is ACTIVE
describe-target-health shows unusedAZ not enabled for the load balancer or target group not used by a listenerEnable the AZ; attach the target group to a listener

Cost and cleanup

Reading and local code are free. Deploying the lab stack costs about USD 0.10 per hour (ALB, three Fargate tasks) ⚠ VERIFY current prices; zonal shift itself has no per-shift charge that we know of ⚠ VERIFY, and zonal autoshift practice runs are best checked against the current ARC pricing page. Destroy the stack within the hour.

# CDK
cd labs/m12-resilient-account-api/infra/cdk && cdk destroy -c owner=<your-name> --profile learn-sandbox
# or Terraform
cd labs/m12-resilient-account-api/infra/terraform && terraform destroy

# Cancel any zonal shift left active (they expire, but do not leave them).
aws arc-zonal-shift list-zonal-shifts --status ACTIVE --profile learn-sandbox --region us-east-1

# Confirm nothing tagged for the lab remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m12-ha \
  --profile learn-sandbox --region us-east-1 --query "ResourceTagMappingList[].ResourceARN"

Exercises

  • 🟢 Guided: Run ReadinessGateTests. Then add /health/live and /health/ready to a small ASP.NET Core app, start it, and call both. Send Ctrl+C and watch /health/ready flip to 503 before the process exits.
  • 🟡 Challenge: Your check interval is 30 s with an unhealthy threshold of 3 and a healthy threshold of 5. Calculate worst-case detection and recovery times, then propose settings for a 30-second detection budget and say what you give up.
  • 🔴 Live problem: "During a Core Banking slowdown, the Accounts API became completely unavailable even though 90% of requests do not need Core Banking." You find the ALB health check calls /health which tests Core Banking and the database. Write the root cause, the fix, and the monitoring you will add so a slow dependency is visible without removing tasks.

Assignment

Create docs/health-check-standard.md for Harbourline: required endpoints, what readiness may and may not check, timing defaults and their reasoning, deregistration and shutdown rules, and a decision table for cross-zone settings. Include a link to your test results.

Solutions

Knowledge check

Interview questions

  • "How do you do zero-downtime deployments behind an ALB?" Readiness that flips to unhealthy on SIGTERM, shutdown shorter than the deregistration delay, rolling or blue/green updates with a minimum-healthy percentage, and a deployment circuit breaker that rolls back.
  • "What goes in a health check?" Liveness checks nothing external. Readiness checks only what makes this task unable to serve right now (draining, still starting, local state). Dependency health belongs in metrics and breakers.
  • "What is fail open?" If no target in a group is healthy, the ALB sends traffic to all of them rather than none, betting that some can still answer. It is why health checks alone cannot protect against a shared failure.
  • "When would you use zonal shift?" When one AZ is impaired in a way health checks do not catch, and the other AZs have the capacity to absorb the traffic. Always set an expiry, and know it will not help in fail-open.

Certification practice

Completion checklist

  • I can explain cross-zone balancing for ALB and NLB and when to turn it off
  • I can write a liveness and a readiness endpoint and explain why readiness has no dependency checks
  • I ran ReadinessGateTests and the drain demo
  • I calculated detection and recovery times for my own health-check settings
  • I ran the cleanup steps and confirmed nothing is still billing

Key takeaways

  • The ALB fails open when all targets are unhealthy, so dependency-aware health checks add risk, not safety.
  • Detection time is interval times unhealthy threshold; keep checks cheap and local.
  • Drain on SIGTERM and keep shutdown shorter than the deregistration delay.
  • Zonal shift is a deliberate, time-limited AZ evacuation that needs spare capacity and does nothing in fail-open.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page