Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: partner API timeouts

Diagnose a partner API that times out by splitting the call into DNS, connect, TLS and response time, checking the egress path and the partner's limits, and choosing timeouts, retries and a circuit breaker that fail safe for a bank.

intermediate⏱ 1 h 20 min🟢 FreeDVA-C03SOA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · Microsoft.Extensions.Http.Resilience · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Break one slow or failing outbound call into DNS, TCP connect, TLS handshake, time to first byte and body transfer, and say which phase is the problem.
  • Walk the egress path from a container in a private subnet to a partner endpoint and name the control or component that can block each hop.
  • Tell a partner problem from an AWS-side problem with evidence you can send to the partner.
  • Choose timeouts, retries and circuit-breaker behaviour that fail safe (for example holding a payment for screening rather than releasing it).
  • Recognise retry amplification and the 429 / Retry-After contract.
  • Run a C# probe that measures every phase of a partner call and use it as evidence in an incident record.

Prerequisites

Why this matters in banking

Harbourline's Payments Service calls a sanctions-screening partner (fictional: api.screening-partner.example) before every outbound payment. At 14:05 the p99 latency of the payment endpoint jumps from 400 ms to 30 seconds and some customers see "Something went wrong". The partner's status page says "all systems operational". The partner support desk says "nothing on our side, check your network". Meanwhile the payment queue grows.

Two lessons come from incidents like this. First, "timeout" is a symptom with at least a dozen causes, and arguing without evidence wastes an hour. Second, the safe failure mode matters more than the fix: a sanctions-screening call that times out must never turn into "assume clean and release the payment". Hold the payment, tell the customer it is delayed, and release only when screening has really answered. Your design decision on what happens when the call fails is a compliance control, so agree it with the compliance team before the incident.

The idea in plain language

A call to a partner is a relay race with five hand-offs, and any one can be slow:

  1. DNS: turn api.screening-partner.example into an IP address.
  2. TCP connect: open a connection (the three-way handshake). Firewalls and routing problems show up here as hangs.
  3. TLS handshake: agree encryption and check certificates (and present a client certificate for mTLS).
  4. Request and wait: send the request and wait for the first byte of the answer. Slow partner processing shows up here.
  5. Response transfer: read the body. Large responses or slow links show up here.

You diagnose by timing each phase separately. A call that takes 30 seconds because DNS answered in 29 seconds is a different incident from one that spent 29 seconds waiting for the partner's first byte.

Analogy: Sending a registered letter to a partner. If the address lookup takes forever (DNS), if the post office will not accept the letter (connect), if the receptionist keeps asking for ID (TLS), if the partner takes a week to reply (response), or if the reply is a 400-page contract (transfer), the customer experience is the same: "no answer". The fix for each is entirely different.

Timeouts versus retries

A timeout says "stop waiting". A retry says "try again". Together they decide how long a caller is blocked and how much extra load you send to a struggling partner. Without a timeout, threads wait on a stuck call forever and your service runs out of threads or connections. With retries but no backoff, you hammer a partner that is already overloaded. With retries at several layers you multiply: three layers each trying three times is nine calls.

What problem the AWS services solve

AWS provides the plumbing and the visibility. It does not choose your timeouts.

  • NAT gateway gives private-subnet workloads a stable set of source IP addresses (its Elastic IPs), which the partner allows through its firewall. It also has limits: 55,000 simultaneous connections per destination IP, port and protocol per address, and a 350-second idle timeout after which it resets the connection (see the NAT lesson).
  • VPC Flow Logs record accepted and rejected network flows, so a security group, network ACL or route problem shows up as REJECT or as no traffic at all.
  • Route 53 Resolver query logs show what your VPC asked DNS and how long it took (if you enable them).
  • AWS Network Firewall or a proxy can allow only listed partner domains; a missing rule is a silent block.
  • CloudWatch, X-Ray and Application Signals show your side of the latency; the NAT gateway publishes ErrorPortAllocation, PacketsDropCount, IdleTimeoutCount and connection metrics.

What AWS cannot do is see inside the partner. Your evidence about the partner is the timing breakdown from your side plus whatever the partner can tell you for the same request.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure NAT Gateway with a static public IPAWS NAT gateway with an Elastic IPSame job. In AWS, a replaced gateway usually gets a new Elastic IP unless you reuse the old one, which breaks partner allow-lists
NSG flow logsVPC Flow Logs (to CloudWatch Logs, S3 or Firehose)Fields differ slightly. REJECT rows show security group or NACL denials
Azure DNS Private Resolver query logsRoute 53 Resolver query loggingOff by default, and costs log storage
Azure Firewall FQDN rulesAWS Network Firewall domain-list rule groupsA TLS SNI or HTTP host rule, so a new partner hostname needs a rule
Polly timeout, retry and circuit breaker policiesMicrosoft.Extensions.Http.Resilience (built on Polly v8)Same ideas in .NET. The AWS SDK has its own separate retry logic as well
Application Insights dependency tracking (duration, success)X-Ray subsegments and OpenTelemetry HTTP client spansYou must enable the instrumentation. Dependency calls are not tracked by default
HttpClient with IHttpClientFactoryThe same, in the same wayThe platform changes (NAT, idle timeouts); the client code does not

Architecture

The egress path and what can block each hop

Each numbered hop has a classic failure:

HopTypical failureHow it looks
1 DNSResolver query logging shows slow or failed lookups. A partner changed its IPs, and a long-lived connection pool still points at the old onesSocketException: Name or service not known, or calls fail only on old connections
2 Security groupOutbound rule missing (the default allows all, but hardened groups restrict it)Connect hangs. Flow Logs show REJECT
3 NACLStateless: return traffic on ephemeral ports (1024 to 65535) blockedConnect hangs or resets. Flow Logs show REJECT in one direction
4 Route tablePrivate subnet has no 0.0.0.0/0 route to the NAT gatewayConnect hangs. No flows leave the VPC
5 NAT / IGWPort exhaustion, idle timeout resets, gateway in a failed AZErrorPortAllocation, IdleTimeoutCount, intermittent resets
6 Partner firewallOur egress IP changed and the partner has not allowed the new oneConnect timeouts only from the new IP. Works from a laptop on the old VPN
TLSPartner certificate expired or chain changed, our client certificate expired, wrong TLS version, SNI mismatchAuthenticationException during the handshake

Which phase is slow?

Step-by-step implementation

Minutes 0 to 5: scope and set the safe mode

  1. How wide? All calls or some? One partner or every dependency? One AZ or all? Compare the error rate by partner, route and Availability Zone in your logs.
  2. Is the failure mode safe? For screening or fraud calls, confirm payments are being held, not released. If a code path fails open, stop it first (feature flag, or pause the worker).
  3. Is it getting worse? If retries are piling up (queue depth, thread count), open the circuit manually or reduce concurrency so the partner can recover. See Retry strategies on retry storms.
  4. Message the partner's support with the time window (in UTC), the endpoint, your source IP (the NAT Elastic IP) and one or two correlation IDs. Ask for their request ID for those calls.

Minutes 5 to 25: time the phases from inside the VPC

A laptop test proves nothing about the VPC path. Test from the same subnet and security group as the failing workload: ECS Exec into a running task, or run the probe below as a one-off task. Compare phases with a known-good baseline (last week's numbers in your runbook).

Minutes 25 to 45: check each hop

Work down the table in the architecture section. Evidence checklist:

  • Flow Logs: any REJECT from the task's ENI to the partner IP?
  • NAT gateway metrics: ErrorPortAllocation, PacketsDropCount, IdleTimeoutCount.
  • Has the NAT gateway or its Elastic IP changed recently (CloudTrail CreateNatGateway, ReleaseAddress)?
  • Certificate details from the probe: expiry date of the partner's certificate and of your client certificate.
  • Your own side: HTTP client metrics (open connections, request queue time), thread-pool starvation, container CPU.

Minutes 45 to 60: decide what to change

FindingChange
Partner first-byte latency is highKeep fail-safe behaviour, reduce concurrency, open the circuit, escalate to the partner with timings
New NAT IP not allowedAsk the partner to allow it, or reattach the original Elastic IP (change control applies)
Port exhaustionReuse connections (singleton HttpClient), add Elastic IPs to the NAT gateway (each adds capacity)
Idle connection reset after 350 secondsSet the pooled idle timeout below 350 seconds, or enable TCP keep-alive
Certificate expiredRotate it, and add an expiry alarm. See Certificates, TLS and mTLS
We retry too muchRemove duplicate retry layers, add backoff with jitter and honour Retry-After

After recovery: prevent and record

Record the phase timings (before and during), the hop that failed, the decision on fail-safe behaviour and the partner correspondence. Add: a synthetic canary that calls the partner health endpoint every minute, a TLS expiry alarm, and a dashboard of partner call latency by phase.

AWS Console instructions

  1. Console → VPC → Your VPCs → Flow logs to confirm flow logs exist for the VPC or subnet. Then CloudWatch → Logs → Logs Insights, select the flow log group and run the query in the CLI section.
  2. Console → VPC → NAT gateways → (your gateway) → Monitoring for ErrorPortAllocation, IdleTimeoutCount and connection counts. The details tab shows the Elastic IP(s) the partner must allow.
  3. Console → ECS → Clusters → (cluster) → Tasks → (task) → Logs / Configuration to check the task's subnet and security group. ECS Exec is enabled on the service (enableExecuteCommand), and you connect with the CLI command below.
  4. Console → Route 53 → Resolver → Query logging (if enabled) to see DNS timings and answers.
  5. Console → CloudWatch → Application Signals or X-Ray traces to find the slow dependency span.

AWS CLI

# 1. Open a shell in a running task (ECS Exec must be enabled; the Session Manager plugin must be installed).
aws ecs execute-command --cluster hl-payments --task <task-id> --container app \
  --interactive --command "/bin/sh" --profile learn-sandbox

# 2. Inside the task: time every phase of one call with curl (if the image includes curl).
curl -sS -o /dev/null \
  -w "dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s code=%{http_code}\n" \
  https://api.screening-partner.example/v1/health

# 3. Find rejected flows to the partner IP in VPC Flow Logs (Logs Insights query against the flow log group).
aws logs start-query --log-group-name /hl/vpc-flow-logs \
  --start-time $(date -d '3 hours ago' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, srcAddr, dstAddr, dstPort, action, protocol
    | filter dstAddr = "203.0.113.25" and dstPort = 443
    | stats count(*) as flows by action, bin(5m)
    | sort @timestamp desc' \
  --profile learn-sandbox

# 4. NAT gateway health: port allocation errors and idle timeouts.
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name ErrorPortAllocation \
  --dimensions Name=NatGatewayId,Value=nat-0123456789abcdef0 \
  --start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum --profile learn-sandbox

aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name IdleTimeoutCount \
  --dimensions Name=NatGatewayId,Value=nat-0123456789abcdef0 \
  --start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum --profile learn-sandbox

# 5. Which Elastic IP(s) must the partner allow? Confirm they have not changed.
aws ec2 describe-nat-gateways --nat-gateway-ids nat-0123456789abcdef0 \
  --query 'NatGateways[].NatGatewayAddresses[].[PublicIp,AllocationId,NetworkInterfaceId]' \
  --output table --profile learn-sandbox

# 6. Who changed the NAT gateway or addresses recently?
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateNatGateway \
  --max-results 10 --profile learn-sandbox

The address 203.0.113.25 is from a documentation range reserved for examples (TEST-NET-3), so it is safe as fictional data.

A Logs Insights query for your own application logs, grouping slow partner calls by phase (if your client logs phase timings):

fields @timestamp, partner, dnsMs, connectMs, tlsMs, ttfbMs, totalMs, outcome
| filter partner = "screening"
| stats pct(totalMs, 99) as p99, pct(ttfbMs, 99) as ttfb99, pct(connectMs, 99) as connect99, count(*) as calls by bin(5m)
| sort @timestamp desc

Infrastructure as Code

The defensive settings belong in code, not in someone's memory. This shows the HTTP client settings and, for the network side, a CloudWatch alarm on NAT port allocation errors.

using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Constructs;

public class PartnerEgressAlarmsStack : Stack
{
    public PartnerEgressAlarmsStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var portErrors = new Metric(new MetricProps
        {
            Namespace = "AWS/NATGateway",
            MetricName = "ErrorPortAllocation",
            DimensionsMap = new Dictionary<string, string> { ["NatGatewayId"] = "nat-0123456789abcdef0" },
            Statistic = "Sum",
            Period = Duration.Minutes(5)
        });

        new Alarm(this, "NatPortAllocation", new AlarmProps
        {
            AlarmName = "hl-nat-port-allocation-errors",
            AlarmDescription = "NAT gateway cannot allocate ports. Check connection reuse, see partner API playbook.",
            Metric = portErrors,
            Threshold = 0,
            EvaluationPeriods = 2,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-partner");
    }
}

Complete code

A probe that times every phase

Run this from inside the failing subnet (for example as a one-off ECS task or through ECS Exec with the .NET runtime image). It uses only the base class library.

PartnerProbe/Program.cs
using System.Diagnostics;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Cryptography.X509Certificates;

var host = args.Length > 0 ? args[0] : "example.com";
var port = args.Length > 1 ? int.Parse(args[1]) : 443;
var path = args.Length > 2 ? args[2] : "/";
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(20));   // overall probe deadline
var sw = Stopwatch.StartNew();

static void Report(string phase, TimeSpan t, string note = "") =>
    Console.WriteLine($"{phase,-10} {t.TotalMilliseconds,8:F0} ms  {note}");

try
{
    // 1. DNS
    var t0 = sw.Elapsed;
    var addresses = await Dns.GetHostAddressesAsync(host, cts.Token);
    Report("dns", sw.Elapsed - t0, string.Join(", ", addresses.Select(a => a.ToString())));

    // 2. TCP connect (first address)
    t0 = sw.Elapsed;
    using var tcp = new TcpClient();
    await tcp.ConnectAsync(addresses[0], port, cts.Token);
    Report("connect", sw.Elapsed - t0, $"to {addresses[0]}:{port}");

    // 3. TLS handshake
    t0 = sw.Elapsed;
    using var ssl = new SslStream(tcp.GetStream(), leaveInnerStreamOpen: false);
    await ssl.AuthenticateAsClientAsync(new SslClientAuthenticationOptions { TargetHost = host }, cts.Token);
    var cert = new X509Certificate2(ssl.RemoteCertificate!);
    Report("tls", sw.Elapsed - t0,
        $"{ssl.SslProtocol}, subject '{cert.Subject}', expires {cert.NotAfter:yyyy-MM-dd} (in {(cert.NotAfter - DateTime.Now).Days} days)");

    // 4. Request and first byte
    t0 = sw.Elapsed;
    var request = $"GET {path} HTTP/1.1\r\nHost: {host}\r\nConnection: close\r\nUser-Agent: hl-probe\r\n\r\n";
    await ssl.WriteAsync(System.Text.Encoding.ASCII.GetBytes(request), cts.Token);
    await ssl.FlushAsync(cts.Token);
    var buffer = new byte[4096];
    var read = await ssl.ReadAsync(buffer, cts.Token);
    Report("ttfb", sw.Elapsed - t0, System.Text.Encoding.ASCII.GetString(buffer, 0, Math.Min(read, 40)).Split('\n')[0].Trim());

    // 5. Drain the body
    t0 = sw.Elapsed;
    long total = read;
    while ((read = await ssl.ReadAsync(buffer, cts.Token)) > 0) total += read;
    Report("body", sw.Elapsed - t0, $"{total} bytes");
    Report("total", sw.Elapsed);
}
catch (OperationCanceledException)
{
    Console.WriteLine($"TIMEOUT after {sw.Elapsed.TotalSeconds:F1}s: the phase printed last is the one BEFORE the stall.");
    return 2;
}
catch (Exception ex)
{
    Console.WriteLine($"FAILED after {sw.Elapsed.TotalMilliseconds:F0} ms: {ex.GetType().Name}: {ex.Message}");
    return 1;
}
return 0;

Run it with dotnet run -- api.screening-partner.example 443 /v1/health. The project needs only the default console template targeting net10.0.

A client configured for a NAT gateway

PartnerClientRegistration.cs
using System.Net.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;

public static class PartnerClientRegistration
{
    public static IServiceCollection AddScreeningClient(this IServiceCollection services)
    {
        services.AddHttpClient("screening", c =>
            {
                c.BaseAddress = new Uri("https://api.screening-partner.example/");
                c.Timeout = Timeout.InfiniteTimeSpan;       // the resilience handler owns timeouts
            })
            .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
            {
                ConnectTimeout = TimeSpan.FromSeconds(3),                 // fail fast on blocked paths
                PooledConnectionLifetime = TimeSpan.FromMinutes(2),       // re-resolve DNS regularly
                PooledConnectionIdleTimeout = TimeSpan.FromSeconds(240),  // under the NAT 350 s idle timeout
                KeepAlivePingDelay = TimeSpan.FromSeconds(60),            // used for HTTP/2 connections
                KeepAlivePingTimeout = TimeSpan.FromSeconds(10)
            })
            .AddStandardResilienceHandler(o =>
            {
                o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(8);
                o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
                o.Retry.MaxRetryAttempts = 2;               // ONE retry layer, not one per component
                o.Retry.UseJitter = true;
                o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);   // at least twice the attempt timeout
            });
        return services;
    }
}

Code explained line by line

  1. The probe prints each phase on its own line. When a call stalls, the last printed phase tells you the stall happened in the next one: a printed dns and no connect means the connect hung, which points to the network path or partner allow-list.
  2. CancellationTokenSource(TimeSpan.FromSeconds(20)) is a hard deadline for the whole probe so the tool itself cannot hang.
  3. Dns.GetHostAddressesAsync shows every address returned. If the partner has several IPs and only one is blocked, this and the connect line reveal it.
  4. AuthenticateAsClientAsync with TargetHost sets SNI, and the certificate line prints the negotiated protocol and the expiry in days: a one-glance certificate health check.
  5. The raw HTTP request avoids HttpClient on purpose, so you measure the network and TLS without any pooling, retry or proxy behaviour of your application.
  6. ConnectTimeout of 3 seconds turns a blackholed path into a quick failure instead of waiting for the operating system's much longer default.
  7. PooledConnectionLifetime forces connections to be replaced regularly, so a changed DNS answer is picked up. Without it, a long-lived HttpClient can keep using an old IP for days.
  8. PooledConnectionIdleTimeout below 350 seconds means your client closes idle connections before the NAT gateway silently resets them, which removes a class of "first call after quiet period fails" errors.
  9. c.Timeout = Timeout.InfiniteTimeSpan hands timeout control to the resilience handler (attempt timeout 2 s inside, total 8 s outside), which is the pattern in the partner API lesson.
  10. One retry layer only: the AWS SDK clients have their own retries (see Retry strategies), so do not stack a third layer in a queue consumer without counting the multiplication.

Security considerations

⚠ Learning shortcut — not for production

The probe uses the default certificate validation, which is correct. During an incident people are tempted to "just ignore the TLS error" (ServerCertificateCustomValidationCallback = (_, _, _, _) => true) to see whether the call works. Never ship or even deploy that. It disables the protection that stops a man-in-the-middle from reading payment data. Use the probe's certificate details to prove the cause, then fix the certificate or trust store properly.

  • Fail closed for controls. A screening or fraud timeout must result in a held payment and a retry later, never an automatic pass. Write this in the design document and test it.
  • Keep the NAT allow-list under change control. Replacing a NAT gateway can break a partner's firewall rule. Treat Elastic IPs as part of the partner contract and reuse them, or notify the partner before a change.
  • mTLS client certificates expire. Alarm on expiry well before the date, and keep the certificate in Secrets Manager, not in the image.
  • Do not paste credentials or full payloads into partner tickets. Send timestamps, correlation IDs, IPs and the timing breakdown.
  • Egress control stays on. While diagnosing, do not widen outbound security group or Network Firewall rules to "allow all". Add the specific destination and record the change. See Segmentation and egress.
  • Nothing here substitutes for your third-party risk process; see Operational resilience and third-party risk.

Banking use case

For Harbourline, the screening call sits in the payment path with a business rule: "no screening result, no release". The Payments Service writes the payment as screening, calls the partner with an 8-second total budget and, on timeout or an open circuit, leaves the payment in screening and puts a retry message on an SQS queue with a delay. The customer sees "Payment pending". A dashboard shows partner latency by phase, the circuit state and the number of payments held. In the 14:05 incident, the phase timings showed DNS and connect at normal speed but time-to-first-byte of 25 seconds, so the team could tell the partner with evidence, reduced concurrency to avoid piling on, and kept payments safe while the partner fixed an overloaded database on their side.

Common mistakes

  • No connect timeout. Blackholed paths wait for the operating system default (which can be minutes), exhausting threads.
  • A new HttpClient per request. Causes port exhaustion on the NAT gateway and repeated TLS handshakes. Use IHttpClientFactory or a singleton.
  • Retry at every layer. SDK retries, resilience-handler retries and queue redelivery multiply. Decide who owns retries.
  • Retrying non-idempotent calls without a key. A retried payment submission is the duplicate payment playbook again.
  • Testing from a laptop. The laptop has a different path and IP. Always test from the workload's subnet.
  • Ignoring Retry-After on 429. Retrying immediately makes you a bad neighbour and gets you blocked for longer.
  • Failing open. Treating a timeout as "no match" in a screening flow.
  • Forgetting the NAT IP is a contract. A new gateway means a new IP, which the partner must allow.

Troubleshooting

SymptomLikely causeFix
Connect hangs, Flow Logs show REJECTSecurity group or NACL blocks outbound or return trafficAdd the specific rule (NACLs are stateless: allow ephemeral ports back)
Connect hangs, no flow records at allNo route to the NAT gateway, or the gateway is in a failed AZCheck the route table and NAT gateway state, use a NAT gateway per AZ
Works from one AZ, times out from anotherDifferent NAT gateway (and a different, non-allow-listed IP) per AZAsk the partner to allow every NAT IP, or use shared addresses by design
ErrorPortAllocation greater than zeroToo many connections to one destination per NAT addressReuse connections, add Elastic IPs, reduce parallelism
First call after a quiet period fails, retry succeedsNAT idle timeout reset a pooled connectionLower pooled idle timeout under 350 seconds, enable keep-alives
AuthenticationException in the TLS phasePartner or client certificate expired, wrong chain, or wrong host nameInspect with the probe or openssl s_client, rotate, fix the trust store
Calls fail only for a day after a partner changeStale DNS in long-lived connectionsSet PooledConnectionLifetime, restart tasks if urgent
HTTP 429Partner rate limit reachedHonour Retry-After, reduce concurrency, ask for a higher limit
Timeouts rise when load risesOur thread pool or connection pool is starved, not the partnerCheck pool metrics and request queue time, cap concurrency with a bulkhead
Only some payments time out, large onesLarge body or slow client upload through a constrained pathCompare body size, check the transfer phase

Cost and cleanup

Cost label: free. The probe runs anywhere. The optional sandbox exercise (a Fargate task in a private subnet behind a NAT gateway) is paid: a NAT gateway costs roughly a few cents per hour plus data processing (⚠ VERIFY current price), so do it only as deploy-and-destroy within an hour, or use the free design-only variant below.

# If you created a sandbox NAT gateway and Elastic IP for this exercise
aws ec2 delete-nat-gateway --nat-gateway-id <nat-id> --profile learn-sandbox
# wait until the state is "deleted", then release the address
aws ec2 release-address --allocation-id <eipalloc-id> --profile learn-sandbox

# Remove the alarm if you created it
aws cloudwatch delete-alarms --alarm-names hl-nat-port-allocation-errors --profile learn-sandbox

# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-partner --profile learn-sandbox

Exercises

  • 🟢 Guided: Run the probe against example.com on port 443 from your machine. Read all five lines. Then run it against a port that is not listening on a host you control (for example a local port) and describe which line is the last one printed and why.
  • 🟡 Challenge: Extend the probe to run ten times and print the p50 and maximum for each phase. Then add a --resolve option that forces a specific IP, so you can test each address the DNS answer returned.
  • 🔴 Live problem: "Calls to the partner started timing out at 14:05 from Availability Zone B only. Zone A is fine. The partner says they see nothing." You have Flow Logs, NAT gateway metrics and CloudTrail. List your first five checks in order, and say what you will send the partner.

Assignment

Create a partner dependency runbook page for your portfolio: a table of phases, the evidence for each, the fail-safe behaviour for one partner, the NAT IPs the partner must allow, the certificate expiry dates and the contact path. Commit it as runbooks/partner-api-timeouts.md with fictional data.

Solutions

Knowledge check

Interview questions

  • A partner API times out intermittently. How do you investigate? Time DNS, connect, TLS, first byte and transfer separately from the same subnet, check Flow Logs and NAT metrics, compare AZs and source IPs, then send the partner evidence.
  • How do you design a call to an unreliable partner? Connect, attempt and total timeouts, bounded jittered retries at one layer only, a circuit breaker, a bulkhead, idempotency keys for state-changing calls and an explicit fail-safe behaviour.
  • What is the danger of a retry storm? Retries multiply across layers and clients and overload the recovering partner, delaying recovery. Backoff, jitter, circuit breakers and honouring Retry-After limit it.
  • How do you handle a partner's rate limit? Respect 429 and Retry-After, cap concurrency, queue work and agree limits in the contract.
  • Why does a NAT gateway matter to a partner integration? Its Elastic IP is the identity the partner allows, and it has connection and idle-timeout limits that affect how you pool connections.

Certification practice

Completion checklist

  • I can name the five phases of an outbound call and what each failure looks like
  • I can list the hops from a private subnet to a partner and one blocker for each
  • I know why a screening timeout must fail closed
  • I ran the probe and read the output
  • I can explain the NAT idle timeout and port exhaustion limits and the client settings that avoid them
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-partner remains

Key takeaways

  • Time each phase (DNS, connect, TLS, first byte, transfer) from inside the failing subnet.
  • Per-AZ or per-IP patterns point to the NAT gateway and the partner's allow-list.
  • Own retries in one layer, with attempt and total timeouts, backoff with jitter and a circuit breaker.
  • Decide the fail-safe behaviour before the incident: for controls, fail closed.
  • Never disable TLS validation to "test"; use the evidence to fix the real cause.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page