Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: 502, 503 and 504 from an ALB or API Gateway

Work out which hop returned a gateway error, read the load balancer and API Gateway evidence, and fix target health, keep-alive, idle-timeout and integration-timeout mismatches without guessing.

intermediate⏱ 1 h 15 min🟢 FreeSOA-C03DVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · Application Load Balancer · API Gateway REST and HTTP APIs · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Decide which hop generated a 502, 503 or 504: the client's edge, the load balancer or gateway, or your own application.
  • Read ALB access-log fields and CloudWatch metrics to separate "the load balancer said no" from "the target said no".
  • Map each gateway error code to its short list of real causes, and rank them for the symptom in front of you.
  • Fix the classic .NET mismatches: Kestrel keep-alive against the ALB idle timeout, health-check path and port, deregistration delay during deployments.
  • Diagnose API Gateway 502 (bad integration response) and 504 (integration timeout) from execution and access logs.
  • Run a C# tool that summarises ALB access-log lines into an evidence table for an incident record.

Prerequisites

Why this matters in banking

At 09:02 on a Monday, Harbourline's mobile banking app starts showing "Service unavailable" for roughly one request in twenty. The on-call engineer sees a graph of 502s on the Application Load Balancer. The first instinct is "the load balancer is broken". It almost never is. The load balancer is the messenger: it received something unacceptable from, or could not reach, one of your targets, and it is reporting that.

Two failures hurt in a bank. The first is blaming the wrong layer and losing an hour: the platform team says "the application is crashing", the application team says "the ALB is dropping us". The second is the retry side effect: a customer sees a 504 on "Send money", presses the button again, and the payment may have succeeded the first time. Gateway errors are therefore tied to the duplicate payment playbook: an HTTP 504 means "I stopped waiting", not "it did not happen".

The idea in plain language

A gateway is a middle man. The five-digit status code you see tells you what the middle man experienced, not what happened to your customer's request.

CodeThe middle man is sayingTypical reality
502 Bad Gateway"The thing behind me answered with nonsense, or hung up on me."Target closed the connection, reset it, returned a malformed response, or a Lambda returned a bad shape
503 Service Unavailable"I have nobody to send this to."No registered or healthy targets, or the gateway is shedding load
504 Gateway Timeout"I waited and nobody answered in time."Target too slow, a blocked connection path, or a timeout setting lower than the work needs

The most useful question in every case is: who generated this code? The load balancer or API Gateway generated it, or your application generated it and the gateway only passed it along. If your application returns a 502 or 503 itself (a .NET exception handler that returns 503 on purpose), the gateway forwards it, and the evidence is in your own logs.

Analogy: A hotel front desk phones a guest's room. "No answer" can mean nobody is in the room (no healthy target), the line is dead (connection failed), the guest picked up and hung up mid-sentence (connection closed), or the guest is still talking after the desk's patience ran out (timeout). The desk's report to the caller sounds the same each time. You find the truth by asking the desk for its call log.

What problem the AWS services solve

An Application Load Balancer (ALB) spreads HTTP traffic across targets, checks their health and records what happened. API Gateway fronts HTTP backends and Lambda functions and does the same for APIs. Both give you three kinds of evidence:

  • Metrics: counts of load-balancer-generated errors separate from target-generated errors.
  • Logs: per-request records saying how long each phase took and what the target said.
  • Health data: which targets are in service, and why one is not.

The limits are honest ones. The ALB's timeouts are generic settings that do not know your business operation takes 40 seconds. API Gateway has an integration timeout ceiling (see the table below). Neither can tell you why your application was slow. They tell you where to look.

The numbers that matter (⚠ VERIFY quotas before relying on them)

SettingValueSource and notes
ALB idle timeoutDefault 60 seconds, configurableThe load balancer closes a connection with no data for this long
ALB connect timeout to a target10 secondsFailure to connect within it gives a 504
ALB SSL handshake timeout to a target10 secondsApplies for HTTPS target groups
ALB with a Lambda targetRequest and response body limit 1 MBA larger Lambda response gives a 502; a larger request gives a 413
ALB response header limit from a target32 KB for the whole headerLarger gives a 502
Kestrel KeepAliveTimeoutDefault 130 secondsLonger than the ALB default, which is the safe direction
API Gateway REST API integration timeoutDefault 29 secondsRaising it for Regional and private REST APIs is possible through a quota request, with possible throttle-limit trade-offs (⚠ VERIFY current rules)
API Gateway HTTP API integration timeoutUp to 30 secondsHard limit (⚠ VERIFY)

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Application Gateway 502 "Bad gateway"ALB 502Same meaning. Application Gateway gives backend health diagnostics, ALB gives the target health reason code
Application Gateway 504 request timeout setting (default 20 seconds)ALB idle timeout (default 60 seconds) and API Gateway integration timeoutOn ALB the same value governs idle waiting, which is how long a target may be silent
Backend health probeALB target group health checkDefines the path, port, success codes, interval and thresholds. Unhealthy targets leave rotation
Front Door / Application Gateway access logs and diagnosticsALB access logs (S3) and API Gateway access logs (CloudWatch Logs)ALB access logs are off by default and go to S3; enable them before you need them
APIM backend timeoutAPI Gateway integration timeoutNot interchangeable. Gateway limits are quotas, not just policy settings
IIS idle time-out and keep-aliveKestrel KeepAliveTimeoutIf the app closes idle connections sooner than the ALB, the ALB can send on a dead connection and get a 502
Application Insights failures and dependency viewCloudWatch metrics, X-Ray or Application Signals tracesCorrelate by trace ID, which the ALB adds as X-Amzn-Trace-Id

Architecture

Where each code can be generated

Decision flow for the first five minutes

Step-by-step implementation

Minutes 0 to 5: scope and protect

  1. Ask: all requests, or some? One path, one Availability Zone, one target, one client type (mobile, partner)? Break the 5xx count down by path and by target in your logs.
  2. If it began within minutes of a deployment, check whether it is only during the deployment window (targets draining) or continuing after it.
  3. Remember the payment trap: if the failing endpoint changes state, warn support that customers who saw a 504 may have succeeded. Idempotency keys (Idempotency keys) make a retry safe.
  4. Decide if the quickest safe mitigation is a rollback, a scale-out, or removing one bad target from rotation. Record the decision and time.

Minutes 5 to 15: which counter moved?

Open the ALB's CloudWatch metrics for the last three hours:

  • HTTPCode_ELB_5XX_Count and the specific HTTPCode_ELB_502_Count, HTTPCode_ELB_503_Count, HTTPCode_ELB_504_Count: generated by the load balancer.
  • HTTPCode_Target_5XX_Count: generated by your application and passed through.
  • HealthyHostCount and UnHealthyHostCount per target group: did capacity vanish?
  • TargetResponseTime: did latency rise before the errors?
  • TargetConnectionErrorCount: failures to even connect to targets.
  • RejectedConnectionCount (if present for your setup): the load balancer hit a connection limit.

Different counters point to different branches of the decision flow above.

Minutes 15 to 35: read the per-request evidence

Enable ALB access logs if they are off (they take effect for new requests only), then query them with Athena, or read the raw files for a short window. The two fields that settle most arguments:

FieldMeaningWhat it tells you
elb_status_codeWhat the load balancer returned to the clientThe 502, 503 or 504 the customer saw
target_status_codeWhat the target returned to the load balancer- means the target never produced a status
target_processing_timeSeconds the target took-1 means no response completed
target:portWhich target handled it- means no target was chosen (503 territory)
error_reasonA short reason code in newer log formatsOften names the exact failure

If elb_status_code is 502 and target_status_code is -, the target never gave a valid response: reset, closed or never connected. If both are 502, your application returned 502 and the ALB passed it on. If elb_status_code is 504 and target_processing_time is -1, the target was too slow or unreachable.

Minutes 35 to 55: test the common causes in order

502 causes, most likely first

  1. Keep-alive mismatch. The target closes an idle connection before the ALB's idle timeout, so the ALB sends a request on a connection the target just closed. Kestrel's 130-second default is already above 60 seconds, but a sidecar, reverse proxy or non-.NET runtime in front of the app may use 5 seconds. Set the target's keep-alive above the ALB idle timeout.
  2. Deployment drain. A target was deregistered while handling a long request and the deregistration delay expired. Compare the error timestamps with the deployment times, and raise the delay or shorten the longest request.
  3. Crash or out-of-memory kill. The container died mid-request. Check ECS stopped-task reasons (ECS task keeps restarting).
  4. Malformed response. Headers over 32 KB, an invalid header, or a Lambda target with a response over 1 MB.
  5. Lambda target errors. Function error, throttling or timeout. Check the Lambda logs.
  6. TLS to the target fails. A re-encrypted HTTPS target group with an expired or mismatched certificate on the target.

503 causes

  1. No healthy targets. Look at HealthyHostCount and the target health reason codes. A bad health-check path after a release fails every new target at once.
  2. Empty target group. An ECS service with desired count zero, or a service that failed to register targets.
  3. A rule pointing at the wrong target group. A listener rule edited in the console sends traffic to a group with no targets.

504 causes

  1. Target slow. TargetResponseTime climbed first. Database lock, thread-pool starvation, a slow partner call (Partner API timeouts).
  2. Blocked path. Security group on the targets does not allow the load balancer, or the network ACL blocks return traffic on ephemeral ports 1024 to 65535.
  3. Timeout mismatch. The operation legitimately takes longer than the idle timeout. Move long work to an asynchronous pattern instead of raising the timeout.
  4. Content-Length mismatch. The target states a larger body than it sends, so the ALB waits for missing bytes.

Minutes 55 to 75: fix, verify, prevent

Apply the smallest change that removes the cause, confirm the error counters return to baseline, and add the guard that would have caught it: an alarm on HTTPCode_ELB_5XX_Count, an alarm on UnHealthyHostCount, a synthetic canary on a health URL, or a deployment circuit breaker. Record the timeline, the evidence and the change number for audit.

AWS Console instructions

  1. Console → EC2 → Load Balancers → (your ALB) → Monitoring for the 5xx and target-response-time graphs. Compare ELB-generated and target-generated lines.
  2. Console → EC2 → Target Groups → (group) → Targets to see each target's state and the health status details (reason code such as "Health checks failed with these codes" or "Request timed out"). The Resource map tab shows unhealthy targets visually.
  3. Console → EC2 → Load Balancers → Attributes to see the idle timeout and whether access logs are on.
  4. Console → API Gateway → (API) → Stages → Logs and tracing to confirm access logging and execution logging. Use Test on a method to see the execution log of one invocation.
  5. Console → CloudWatch → Logs → Logs Insights for the API Gateway access log group and your application log group.

AWS CLI

# 1. Target health with reason codes for one target group.
aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/hl-payments/0123456789abcdef \
  --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]' \
  --output table --profile learn-sandbox

# 2. ELB-generated 5xx, five-minute buckets, last three hours. The LoadBalancer value is the part of the ARN after "loadbalancer/".
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/hl-public/0123456789abcdef \
  --start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum --profile learn-sandbox

# 3. Target-generated 5xx for the same window (compare with the line above).
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/hl-public/0123456789abcdef \
  --start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum --profile learn-sandbox

# 4. Load balancer attributes: idle timeout and access-log settings.
aws elbv2 describe-load-balancer-attributes \
  --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/app/hl-public/0123456789abcdef \
  --query 'Attributes[?Key==`idle_timeout.timeout_seconds` || Key==`access_logs.s3.enabled`]' \
  --profile learn-sandbox

# 5. Target group attributes: deregistration delay (drain time).
aws elbv2 describe-target-group-attributes \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/hl-payments/0123456789abcdef \
  --query 'Attributes[?Key==`deregistration_delay.timeout_seconds`]' --profile learn-sandbox

# 6. Did anyone change the listener rules or target group recently?
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=elasticloadbalancing.amazonaws.com \
  --max-results 20 --query 'Events[].[EventTime,EventName,Username]' --output table --profile learn-sandbox

API Gateway access-log query, assuming the log format below (Infrastructure as Code section) writes JSON:

fields @timestamp, status, integrationStatus, integrationLatency, responseLatency, errorMessage, requestId
| filter status >= 500
| stats count(*) as errors, avg(integrationLatency) as avgIntegration, max(integrationLatency) as maxIntegration by status, integrationStatus, errorMessage
| sort errors desc

Infrastructure as Code

The settings that prevent these incidents belong in code: explicit health check, drain time, idle timeout, and an API Gateway access log that carries the fields you need during diagnosis.

using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.EC2;
using Amazon.CDK.AWS.ElasticLoadBalancingV2;
using Constructs;

public class GatewayHealthStack : Stack
{
    public GatewayHealthStack(Construct scope, string id, IStackProps props, IVpc vpc) : base(scope, id, props)
    {
        var alb = new ApplicationLoadBalancer(this, "PublicAlb", new ApplicationLoadBalancerProps
        {
            Vpc = vpc,
            InternetFacing = true,
            IdleTimeout = Duration.Seconds(60),   // keep the app keep-alive above this value
            DropInvalidHeaderFields = true
        });

        var targets = new ApplicationTargetGroup(this, "PaymentsTargets", new ApplicationTargetGroupProps
        {
            Vpc = vpc,
            Port = 8080,
            Protocol = ApplicationProtocol.HTTP,
            TargetType = TargetType.IP,
            DeregistrationDelay = Duration.Seconds(60),   // longer than your longest normal request
            HealthCheck = new Amazon.CDK.AWS.ElasticLoadBalancingV2.HealthCheck
            {
                Path = "/health/ready",
                Port = "8080",
                HealthyHttpCodes = "200",
                Interval = Duration.Seconds(15),
                Timeout = Duration.Seconds(5),
                HealthyThresholdCount = 2,
                UnhealthyThresholdCount = 3
            }
        });

        // Alarm on errors the load balancer itself generated.
        new Alarm(this, "Elb5xx", new AlarmProps
        {
            AlarmName = "hl-alb-elb-5xx",
            AlarmDescription = "ALB generated 5xx. See playbook alb-api-gateway-502-503-504.",
            Metric = alb.Metrics.HttpCodeElb(HttpCodeElb.ELB_5XX_COUNT, new MetricOptions
            {
                Period = Duration.Minutes(1),
                Statistic = "Sum"
            }),
            Threshold = 5,
            EvaluationPeriods = 3,
            DatapointsToAlarm = 2,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        // Alarm when any target is unhealthy.
        new Alarm(this, "UnhealthyTargets", new AlarmProps
        {
            AlarmName = "hl-payments-unhealthy-targets",
            Metric = targets.Metrics.UnhealthyHostCount(new MetricOptions { Period = Duration.Minutes(1) }),
            Threshold = 0,
            EvaluationPeriods = 3,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-gateway-errors");
    }
}

The API Gateway access-log format (JSON) that makes the queries above work, set on the stage's accessLogSettings:

{"requestId":"$context.requestId","extendedRequestId":"$context.extendedRequestId","status":"$context.status","integrationStatus":"$context.integrationStatus","integrationLatency":"$context.integrationLatency","responseLatency":"$context.responseLatency","errorMessage":"$context.error.message","errorType":"$context.error.responseType","path":"$context.path"}

Complete code

A tool that turns ALB access-log lines into an evidence table

ALB access logs are space-separated with quoted fields. This tool reads a log file (plain text, already unzipped) and counts requests by the ALB's status, the target's status and the target. It needs only the base class library. The field order used here is the documented start of the ALB log format: type, time, elb, client, target, three processing times, elb_status_code, target_status_code, byte counts, then the quoted request (⚠ VERIFY against the current log-entry documentation before relying on column positions beyond these).

AlbLogSummary/Program.cs
using System.Text;

if (args.Length == 0)
{
    Console.WriteLine("usage: AlbLogSummary <access-log-file>");
    return 1;
}

var groups = new Dictionary<(string Elb, string Target, string TargetHost), (int Count, double MaxTargetSeconds)>();
var total = 0;

foreach (var line in File.ReadLines(args[0]))
{
    var f = SplitLogLine(line);
    if (f.Count < 13) continue;                       // not a request line

    var elbStatus = f[8];                             // what the client saw
    var targetStatus = f[9];                          // what the target said; "-" means none
    var targetHost = f[4];                            // ip:port or "-"
    _ = double.TryParse(f[6], System.Globalization.CultureInfo.InvariantCulture, out var targetSeconds);

    var key = (elbStatus, targetStatus, targetHost);
    groups.TryGetValue(key, out var current);
    groups[key] = (current.Count + 1, Math.Max(current.MaxTargetSeconds, targetSeconds));
    total++;
}

Console.WriteLine($"Requests read: {total}");
Console.WriteLine($"{"elb",-5} {"target",-7} {"target host",-22} {"count",6} {"max target s",12}  reading");

foreach (var (key, value) in groups.OrderByDescending(g => g.Value.Count))
{
    Console.WriteLine($"{key.Elb,-5} {key.Target,-7} {key.TargetHost,-22} {value.Count,6} {value.MaxTargetSeconds,12:F3}  {Explain(key.Elb, key.Target, value.MaxTargetSeconds)}");
}
return 0;

static string Explain(string elb, string target, double maxTargetSeconds) => (elb, target) switch
{
    ("502", "-") => "ALB 502, target gave no valid status: reset, closed or bad response",
    ("502", "502") => "target itself returned 502 (application or sidecar)",
    ("503", "-") => "ALB 503 with no target: nobody healthy or registered",
    ("504", "-") => maxTargetSeconds > 0
        ? "ALB 504: target connected but stayed silent until the timeout"
        : "ALB 504: no response completed, slow or blocked target",
    (_, "500") or (_, "503") => "application-generated 5xx passed through",
    _ => ""
};

// Splits one log line on spaces, keeping "quoted fields" together.
static List<string> SplitLogLine(string line)
{
    var result = new List<string>();
    var sb = new StringBuilder();
    var inQuotes = false;
    foreach (var c in line)
    {
        if (c == '"') { inQuotes = !inQuotes; continue; }
        if (c == ' ' && !inQuotes)
        {
            if (sb.Length > 0) { result.Add(sb.ToString()); sb.Clear(); }
            continue;
        }
        sb.Append(c);
    }
    if (sb.Length > 0) result.Add(sb.ToString());
    return result;
}

Create a console project targeting net10.0, save the code as Program.cs, and run dotnet run -- sample-alb.log. A small fictional sample file for practice:

sample-alb.log
https 2026-11-18T09:02:11.123456Z app/hl-public/0123456789abcdef 192.0.2.10:51234 10.0.1.25:8080 0.001 0.020 0.000 200 200 120 540 "GET https://api.harbourline.example:443/v1/accounts HTTP/2.0" "HarbourlineApp/5.1" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
https 2026-11-18T09:02:12.223456Z app/hl-public/0123456789abcdef 192.0.2.11:51235 10.0.1.25:8080 0.001 -1 -1 502 - 130 0 "POST https://api.harbourline.example:443/v1/payments HTTP/2.0" "HarbourlineApp/5.1" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
https 2026-11-18T09:02:13.323456Z app/hl-public/0123456789abcdef 192.0.2.12:51236 - 0.000 -1 -1 503 - 130 0 "GET https://api.harbourline.example:443/v1/accounts HTTP/2.0" "HarbourlineApp/5.1" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
https 2026-11-18T09:02:14.423456Z app/hl-public/0123456789abcdef 192.0.2.13:51237 10.0.2.31:8080 0.001 60.001 -1 504 - 130 0 "GET https://api.harbourline.example:443/v1/statements HTTP/2.0" "HarbourlineApp/5.1" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

The addresses come from the reserved documentation range 192.0.2.0/24, so they are safe as fictional data.

Kestrel settings that avoid keep-alive 502s

Program.cs (excerpt)
var builder = WebApplication.CreateBuilder(args);

builder.WebHost.ConfigureKestrel(options =>
{
    // Must stay ABOVE the ALB idle timeout (60 s default) so the app never closes a connection first.
    options.Limits.KeepAliveTimeout = TimeSpan.FromSeconds(120);
    // Bound how long a slow client can hold a request open.
    options.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(30);
});

builder.Services.AddHealthChecks();   // readiness endpoint without downstream calls, see the health checks lesson

var app = builder.Build();
app.MapHealthChecks("/health/ready");
app.Run();

A Lambda proxy response that API Gateway accepts

Function.cs
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

public class Function
{
    public APIGatewayProxyResponse Handle(APIGatewayProxyRequest request, ILambdaContext context)
    {
        // The proxy integration requires this shape: statusCode, headers, a STRING body, isBase64Encoded.
        return new APIGatewayProxyResponse
        {
            StatusCode = 200,
            Headers = new Dictionary<string, string> { ["Content-Type"] = "application/json" },
            Body = "{\"status\":\"ok\"}",
            IsBase64Encoded = false
        };
    }
}

Code explained line by line

  1. SplitLogLine walks the line character by character and toggles inQuotes on each double quote, so the request field (which contains spaces) stays as one value. Plain Split(' ') would break it.
  2. f[8] and f[9] are the two status fields at the heart of the playbook. Printing them side by side answers "did the load balancer or the target say this?".
  3. f[6] is the target processing time. A value of -1 means the target never completed a response, which separates "slow" from "never answered".
  4. The switch turns the pairs into plain sentences. The "502"/"-" pair, for example, is the signature of a connection reset or closure, not an application exception.
  5. KeepAliveTimeout set to 120 seconds keeps the app's idle-connection timer longer than the ALB's 60 seconds, so the load balancer is always the side that closes an idle connection.
  6. The health endpoint /health/ready is deliberately kept free of database and partner calls, so a slow downstream does not make every target unhealthy and trigger a 503 for everybody.
  7. APIGatewayProxyResponse has the properties the proxy integration needs. A string Body is the detail people miss: returning an object where a string is expected makes API Gateway answer 502 "Malformed Lambda proxy response".

Security considerations

⚠ Learning shortcut — not for production

The sample probes and the health endpoint here are unauthenticated, and the sample log lines contain client IPs. During an incident people also paste raw access logs into chat and tickets. Access logs contain client IP addresses and request paths, which are personal data under UK GDPR in many contexts. In production, keep the log bucket private with encryption on, restrict who can read it, share aggregated counts in tickets and follow your retention policy.

  • Do not widen security groups to "fix" a 504. Add the specific rule (load balancer security group to the target port) and record it. See Security groups versus NACLs.
  • Keep drop_invalid_header_fields on so malformed headers do not pass through to the app.
  • Health endpoints must not leak internals. Return a status, not connection strings, versions or stack traces.
  • Access logs and WAF: if a WAF blocks requests the ALB returns 403, not 5xx. Do not mistake a WAF block for an application error. See WAF and Shield.
  • Never disable TLS validation between the ALB and targets to get a 502 to stop. Fix the certificate.

Banking use case

Harbourline's payment endpoint sits behind an ALB, with long-running statement exports served through API Gateway. On the Monday incident, HTTPCode_ELB_502_Count rose at 09:02, exactly when a new task definition rolled out. The log tool showed 502 with target status - and the same two targets, both being drained. The deregistration delay was 10 seconds while the payment submission could run 25 seconds against a slow core-banking call. Raising the delay to 60 seconds and shortening the downstream call timeout removed the errors. The audit record states the evidence (counters, log excerpts, deployment times), the change and its approval, and the customer-impact note: 31 payment submissions had seen a 502, and support verified each against the idempotency table instead of asking customers to retry.

Common mistakes

  • Treating all 5xx as one thing. Split ELB-generated from target-generated counts first.
  • Raising the idle timeout to hide a slow operation. It moves the failure to the client or API Gateway. Make long work asynchronous instead.
  • A health check that calls the database. A database blip then makes every target unhealthy and the ALB returns 503 even though the app could still serve cached or partial responses.
  • A health-check path that requires authentication. The ALB gets 401, marks every target unhealthy and returns 503. Allow the health path anonymously.
  • Turning on access logs after the incident. They are not retroactive.
  • Short keep-alive on a proxy or sidecar between the ALB and Kestrel.
  • Forgetting deployments. New tasks that need 60 seconds to start with a 30-second health-check grace period fail checks and cause a flapping 503.
  • Telling customers to retry blindly on a 504 for a money-moving endpoint.

Troubleshooting

SymptomLikely causeFix
502 with target status -, sporadic, low rateTarget closing idle connections before the ALB doesRaise the app or proxy keep-alive above the ALB idle timeout
502 only during deploymentsTargets deregistered while serving requestsRaise deregistration delay, handle SIGTERM gracefully, finish in-flight requests
503 and HealthyHostCount is 0Health check failing on all targetsRead the reason code, fix the path, port or success codes, check target security group
503 with No registered targetsService scaled to zero, or registration failedCheck ECS service events and desired count
504 and TargetResponseTime risingSlow dependency or thread starvation in the appUse traces and app metrics, see Partner API timeouts
504 with no target latency in logsConnection could not be establishedCheck target security group, NACL ephemeral ports, target listening on the port
API Gateway 502 "Malformed Lambda proxy response"Lambda returned the wrong shapeReturn statusCode, headers, string body
API Gateway 504 "Endpoint request timed out"Integration slower than the integration timeoutSpeed up, make asynchronous, or follow the quota process for a longer timeout (⚠ VERIFY)
API Gateway 429 mistaken for an outageThrottlingCheck stage and account throttle settings and the usage plan
460 in the ALB logsClient closed the connection firstAlign the client timeout with the ALB idle timeout

Cost and cleanup

Cost label: free. The log tool and the Kestrel settings run locally. If you build a sandbox ALB to reproduce a 503 (an ALB with an empty target group), that is paid at an hourly rate plus capacity units (⚠ VERIFY current price). Deploy and destroy within an hour, or use the sample log file, which is the free alternative.

# Only if you created a sandbox ALB for this lesson
aws elbv2 delete-load-balancer --load-balancer-arn <alb-arn> --profile learn-sandbox
aws elbv2 delete-target-group --target-group-arn <tg-arn> --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-alb-elb-5xx hl-payments-unhealthy-targets --profile learn-sandbox

# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-gateway-errors --profile learn-sandbox

Exercises

  • 🟢 Guided: Save the four-line sample log, run the tool and write one sentence for each output row saying what happened and which team should look first.
  • 🟡 Challenge: Extend the tool to take a --minutes window, group by minute and show the time at which 502s started. Then add a check that flags any 504 whose target_processing_time is close to the idle timeout you pass in.
  • 🔴 Live problem: "Customers get a 502 on 'Send money' about once an hour, only for requests that were idle for a while. Target logs show nothing." The path is ALB, then a reverse-proxy sidecar container, then Kestrel. List your hypotheses in order, the evidence for each and the single setting change you would try first.

Assignment

Write a gateway-errors runbook page for your portfolio: the decision flow, the evidence table (field, meaning), the three alarms from this lesson and the idle-timeout and keep-alive values you chose with a reason for each. Commit as runbooks/alb-gateway-errors.md with fictional data.

Solutions

Knowledge check

Interview questions

  • Users report intermittent 502s from an ALB. How do you investigate? Split ELB-generated from target-generated 5xx, read access-log status pairs, check for keep-alive mismatch and deployment correlation, and look at target logs and stopped-task reasons.
  • What is the difference between 502, 503 and 504 on an ALB? 502: invalid or no response from the target. 503: no healthy or registered target. 504: connection or response timeout.
  • How do you make a long-running operation work behind API Gateway's timeout? Return 202 with a job ID, process asynchronously (queue plus worker or Step Functions) and let the client poll or receive a notification.
  • How should a health check be designed? Cheap, anonymous, fast, independent of downstream services for liveness, and separate from a deeper readiness check where needed.
  • What would you alarm on? ELB-generated 5xx, unhealthy target count, rising target response time and, for APIs, 5xx and latency by stage.

Certification practice

Completion checklist

  • I can say what 502, 503 and 504 each mean on an ALB and on API Gateway
  • I can tell an ELB-generated 5xx from a target-generated one using metrics and logs
  • I ran the log tool on the sample file and explained every row
  • I know why Kestrel keep-alive must exceed the ALB idle timeout
  • I can explain why a 504 does not prove a payment failed
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-gateway-errors remains

Key takeaways

  • First question: who generated the code? Compare elb_status_code with target_status_code.
  • 502 is a bad or missing response, 503 is no target, 504 is waiting too long.
  • Keep-alive above the idle timeout, drain time above your longest request, health checks cheap and anonymous.
  • A gateway timeout never proves the operation failed: protect retries with idempotency.
  • Turn on access logs before you need them.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page