Playbook: partner API timeouts
Diagnose a partner API that times out by splitting the call into DNS, connect, TLS and response time, checking the egress path and the partner's limits, and choosing timeouts, retries and a circuit breaker that fail safe for a bank.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 · AWS CLI v2 · Microsoft.Extensions.Http.Resilience · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this playbook you can:
- Break one slow or failing outbound call into DNS, TCP connect, TLS handshake, time to first byte and body transfer, and say which phase is the problem.
- Walk the egress path from a container in a private subnet to a partner endpoint and name the control or component that can block each hop.
- Tell a partner problem from an AWS-side problem with evidence you can send to the partner.
- Choose timeouts, retries and circuit-breaker behaviour that fail safe (for example holding a payment for screening rather than releasing it).
- Recognise retry amplification and the 429 /
Retry-Aftercontract. - Run a C# probe that measures every phase of a partner call and use it as evidence in an incident record.
Prerequisites
- Calling partner APIs safely: egress, secrets, timeouts, retries and circuit breakers, which builds the resilient client this playbook diagnoses
- NAT gateways and controlled egress and VPC Flow Logs
- Route 53 and DNS and Certificates, TLS and mTLS
- Retry strategies
- Tools: AWS CLI v2 with the
learn-sandboxprofile; .NET 10 SDK for the probe
Why this matters in banking
Harbourline's Payments Service calls a sanctions-screening partner (fictional: api.screening-partner.example) before every outbound payment. At 14:05 the p99 latency of the payment endpoint jumps from 400 ms to 30 seconds and some customers see "Something went wrong". The partner's status page says "all systems operational". The partner support desk says "nothing on our side, check your network". Meanwhile the payment queue grows.
Two lessons come from incidents like this. First, "timeout" is a symptom with at least a dozen causes, and arguing without evidence wastes an hour. Second, the safe failure mode matters more than the fix: a sanctions-screening call that times out must never turn into "assume clean and release the payment". Hold the payment, tell the customer it is delayed, and release only when screening has really answered. Your design decision on what happens when the call fails is a compliance control, so agree it with the compliance team before the incident.
The idea in plain language
A call to a partner is a relay race with five hand-offs, and any one can be slow:
- DNS: turn
api.screening-partner.exampleinto an IP address. - TCP connect: open a connection (the three-way handshake). Firewalls and routing problems show up here as hangs.
- TLS handshake: agree encryption and check certificates (and present a client certificate for mTLS).
- Request and wait: send the request and wait for the first byte of the answer. Slow partner processing shows up here.
- Response transfer: read the body. Large responses or slow links show up here.
You diagnose by timing each phase separately. A call that takes 30 seconds because DNS answered in 29 seconds is a different incident from one that spent 29 seconds waiting for the partner's first byte.
Analogy: Sending a registered letter to a partner. If the address lookup takes forever (DNS), if the post office will not accept the letter (connect), if the receptionist keeps asking for ID (TLS), if the partner takes a week to reply (response), or if the reply is a 400-page contract (transfer), the customer experience is the same: "no answer". The fix for each is entirely different.
Timeouts versus retries
A timeout says "stop waiting". A retry says "try again". Together they decide how long a caller is blocked and how much extra load you send to a struggling partner. Without a timeout, threads wait on a stuck call forever and your service runs out of threads or connections. With retries but no backoff, you hammer a partner that is already overloaded. With retries at several layers you multiply: three layers each trying three times is nine calls.
What problem the AWS services solve
AWS provides the plumbing and the visibility. It does not choose your timeouts.
- NAT gateway gives private-subnet workloads a stable set of source IP addresses (its Elastic IPs), which the partner allows through its firewall. It also has limits: 55,000 simultaneous connections per destination IP, port and protocol per address, and a 350-second idle timeout after which it resets the connection (see the NAT lesson).
- VPC Flow Logs record accepted and rejected network flows, so a security group, network ACL or route problem shows up as
REJECTor as no traffic at all. - Route 53 Resolver query logs show what your VPC asked DNS and how long it took (if you enable them).
- AWS Network Firewall or a proxy can allow only listed partner domains; a missing rule is a silent block.
- CloudWatch, X-Ray and Application Signals show your side of the latency; the NAT gateway publishes
ErrorPortAllocation,PacketsDropCount,IdleTimeoutCountand connection metrics.
What AWS cannot do is see inside the partner. Your evidence about the partner is the timing breakdown from your side plus whatever the partner can tell you for the same request.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure NAT Gateway with a static public IP | AWS NAT gateway with an Elastic IP | Same job. In AWS, a replaced gateway usually gets a new Elastic IP unless you reuse the old one, which breaks partner allow-lists |
| NSG flow logs | VPC Flow Logs (to CloudWatch Logs, S3 or Firehose) | Fields differ slightly. REJECT rows show security group or NACL denials |
| Azure DNS Private Resolver query logs | Route 53 Resolver query logging | Off by default, and costs log storage |
| Azure Firewall FQDN rules | AWS Network Firewall domain-list rule groups | A TLS SNI or HTTP host rule, so a new partner hostname needs a rule |
| Polly timeout, retry and circuit breaker policies | Microsoft.Extensions.Http.Resilience (built on Polly v8) | Same ideas in .NET. The AWS SDK has its own separate retry logic as well |
| Application Insights dependency tracking (duration, success) | X-Ray subsegments and OpenTelemetry HTTP client spans | You must enable the instrumentation. Dependency calls are not tracked by default |
HttpClient with IHttpClientFactory | The same, in the same way | The platform changes (NAT, idle timeouts); the client code does not |
Architecture
The egress path and what can block each hop
Each numbered hop has a classic failure:
| Hop | Typical failure | How it looks |
|---|---|---|
| 1 DNS | Resolver query logging shows slow or failed lookups. A partner changed its IPs, and a long-lived connection pool still points at the old ones | SocketException: Name or service not known, or calls fail only on old connections |
| 2 Security group | Outbound rule missing (the default allows all, but hardened groups restrict it) | Connect hangs. Flow Logs show REJECT |
| 3 NACL | Stateless: return traffic on ephemeral ports (1024 to 65535) blocked | Connect hangs or resets. Flow Logs show REJECT in one direction |
| 4 Route table | Private subnet has no 0.0.0.0/0 route to the NAT gateway | Connect hangs. No flows leave the VPC |
| 5 NAT / IGW | Port exhaustion, idle timeout resets, gateway in a failed AZ | ErrorPortAllocation, IdleTimeoutCount, intermittent resets |
| 6 Partner firewall | Our egress IP changed and the partner has not allowed the new one | Connect timeouts only from the new IP. Works from a laptop on the old VPN |
| TLS | Partner certificate expired or chain changed, our client certificate expired, wrong TLS version, SNI mismatch | AuthenticationException during the handshake |
Which phase is slow?
Step-by-step implementation
Minutes 0 to 5: scope and set the safe mode
- How wide? All calls or some? One partner or every dependency? One AZ or all? Compare the error rate by partner, route and Availability Zone in your logs.
- Is the failure mode safe? For screening or fraud calls, confirm payments are being held, not released. If a code path fails open, stop it first (feature flag, or pause the worker).
- Is it getting worse? If retries are piling up (queue depth, thread count), open the circuit manually or reduce concurrency so the partner can recover. See Retry strategies on retry storms.
- Message the partner's support with the time window (in UTC), the endpoint, your source IP (the NAT Elastic IP) and one or two correlation IDs. Ask for their request ID for those calls.
Minutes 5 to 25: time the phases from inside the VPC
A laptop test proves nothing about the VPC path. Test from the same subnet and security group as the failing workload: ECS Exec into a running task, or run the probe below as a one-off task. Compare phases with a known-good baseline (last week's numbers in your runbook).
Minutes 25 to 45: check each hop
Work down the table in the architecture section. Evidence checklist:
- Flow Logs: any
REJECTfrom the task's ENI to the partner IP? - NAT gateway metrics:
ErrorPortAllocation,PacketsDropCount,IdleTimeoutCount. - Has the NAT gateway or its Elastic IP changed recently (CloudTrail
CreateNatGateway,ReleaseAddress)? - Certificate details from the probe: expiry date of the partner's certificate and of your client certificate.
- Your own side: HTTP client metrics (open connections, request queue time), thread-pool starvation, container CPU.
Minutes 45 to 60: decide what to change
| Finding | Change |
|---|---|
| Partner first-byte latency is high | Keep fail-safe behaviour, reduce concurrency, open the circuit, escalate to the partner with timings |
| New NAT IP not allowed | Ask the partner to allow it, or reattach the original Elastic IP (change control applies) |
| Port exhaustion | Reuse connections (singleton HttpClient), add Elastic IPs to the NAT gateway (each adds capacity) |
| Idle connection reset after 350 seconds | Set the pooled idle timeout below 350 seconds, or enable TCP keep-alive |
| Certificate expired | Rotate it, and add an expiry alarm. See Certificates, TLS and mTLS |
| We retry too much | Remove duplicate retry layers, add backoff with jitter and honour Retry-After |
After recovery: prevent and record
Record the phase timings (before and during), the hop that failed, the decision on fail-safe behaviour and the partner correspondence. Add: a synthetic canary that calls the partner health endpoint every minute, a TLS expiry alarm, and a dashboard of partner call latency by phase.
AWS Console instructions
- Console → VPC → Your VPCs → Flow logs to confirm flow logs exist for the VPC or subnet. Then CloudWatch → Logs → Logs Insights, select the flow log group and run the query in the CLI section.
- Console → VPC → NAT gateways → (your gateway) → Monitoring for
ErrorPortAllocation,IdleTimeoutCountand connection counts. The details tab shows the Elastic IP(s) the partner must allow. - Console → ECS → Clusters → (cluster) → Tasks → (task) → Logs / Configuration to check the task's subnet and security group. ECS Exec is enabled on the service (
enableExecuteCommand), and you connect with the CLI command below. - Console → Route 53 → Resolver → Query logging (if enabled) to see DNS timings and answers.
- Console → CloudWatch → Application Signals or X-Ray traces to find the slow dependency span.
AWS CLI
# 1. Open a shell in a running task (ECS Exec must be enabled; the Session Manager plugin must be installed).
aws ecs execute-command --cluster hl-payments --task <task-id> --container app \
--interactive --command "/bin/sh" --profile learn-sandbox
# 2. Inside the task: time every phase of one call with curl (if the image includes curl).
curl -sS -o /dev/null \
-w "dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s code=%{http_code}\n" \
https://api.screening-partner.example/v1/health
# 3. Find rejected flows to the partner IP in VPC Flow Logs (Logs Insights query against the flow log group).
aws logs start-query --log-group-name /hl/vpc-flow-logs \
--start-time $(date -d '3 hours ago' +%s) --end-time $(date +%s) \
--query-string 'fields @timestamp, srcAddr, dstAddr, dstPort, action, protocol
| filter dstAddr = "203.0.113.25" and dstPort = 443
| stats count(*) as flows by action, bin(5m)
| sort @timestamp desc' \
--profile learn-sandbox
# 4. NAT gateway health: port allocation errors and idle timeouts.
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name ErrorPortAllocation \
--dimensions Name=NatGatewayId,Value=nat-0123456789abcdef0 \
--start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Sum --profile learn-sandbox
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name IdleTimeoutCount \
--dimensions Name=NatGatewayId,Value=nat-0123456789abcdef0 \
--start-time $(date -u -d '3 hours ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
--period 300 --statistics Sum --profile learn-sandbox
# 5. Which Elastic IP(s) must the partner allow? Confirm they have not changed.
aws ec2 describe-nat-gateways --nat-gateway-ids nat-0123456789abcdef0 \
--query 'NatGateways[].NatGatewayAddresses[].[PublicIp,AllocationId,NetworkInterfaceId]' \
--output table --profile learn-sandbox
# 6. Who changed the NAT gateway or addresses recently?
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateNatGateway \
--max-results 10 --profile learn-sandboxThe address 203.0.113.25 is from a documentation range reserved for examples (TEST-NET-3), so it is safe as fictional data.
A Logs Insights query for your own application logs, grouping slow partner calls by phase (if your client logs phase timings):
fields @timestamp, partner, dnsMs, connectMs, tlsMs, ttfbMs, totalMs, outcome
| filter partner = "screening"
| stats pct(totalMs, 99) as p99, pct(ttfbMs, 99) as ttfb99, pct(connectMs, 99) as connect99, count(*) as calls by bin(5m)
| sort @timestamp descInfrastructure as Code
The defensive settings belong in code, not in someone's memory. This shows the HTTP client settings and, for the network side, a CloudWatch alarm on NAT port allocation errors.
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Constructs;
public class PartnerEgressAlarmsStack : Stack
{
public PartnerEgressAlarmsStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
{
var portErrors = new Metric(new MetricProps
{
Namespace = "AWS/NATGateway",
MetricName = "ErrorPortAllocation",
DimensionsMap = new Dictionary<string, string> { ["NatGatewayId"] = "nat-0123456789abcdef0" },
Statistic = "Sum",
Period = Duration.Minutes(5)
});
new Alarm(this, "NatPortAllocation", new AlarmProps
{
AlarmName = "hl-nat-port-allocation-errors",
AlarmDescription = "NAT gateway cannot allocate ports. Check connection reuse, see partner API playbook.",
Metric = portErrors,
Threshold = 0,
EvaluationPeriods = 2,
ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
TreatMissingData = TreatMissingData.NOT_BREACHING
});
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-partner");
}
}Complete code
A probe that times every phase
Run this from inside the failing subnet (for example as a one-off ECS task or through ECS Exec with the .NET runtime image). It uses only the base class library.
using System.Diagnostics;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Cryptography.X509Certificates;
var host = args.Length > 0 ? args[0] : "example.com";
var port = args.Length > 1 ? int.Parse(args[1]) : 443;
var path = args.Length > 2 ? args[2] : "/";
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(20)); // overall probe deadline
var sw = Stopwatch.StartNew();
static void Report(string phase, TimeSpan t, string note = "") =>
Console.WriteLine($"{phase,-10} {t.TotalMilliseconds,8:F0} ms {note}");
try
{
// 1. DNS
var t0 = sw.Elapsed;
var addresses = await Dns.GetHostAddressesAsync(host, cts.Token);
Report("dns", sw.Elapsed - t0, string.Join(", ", addresses.Select(a => a.ToString())));
// 2. TCP connect (first address)
t0 = sw.Elapsed;
using var tcp = new TcpClient();
await tcp.ConnectAsync(addresses[0], port, cts.Token);
Report("connect", sw.Elapsed - t0, $"to {addresses[0]}:{port}");
// 3. TLS handshake
t0 = sw.Elapsed;
using var ssl = new SslStream(tcp.GetStream(), leaveInnerStreamOpen: false);
await ssl.AuthenticateAsClientAsync(new SslClientAuthenticationOptions { TargetHost = host }, cts.Token);
var cert = new X509Certificate2(ssl.RemoteCertificate!);
Report("tls", sw.Elapsed - t0,
$"{ssl.SslProtocol}, subject '{cert.Subject}', expires {cert.NotAfter:yyyy-MM-dd} (in {(cert.NotAfter - DateTime.Now).Days} days)");
// 4. Request and first byte
t0 = sw.Elapsed;
var request = $"GET {path} HTTP/1.1\r\nHost: {host}\r\nConnection: close\r\nUser-Agent: hl-probe\r\n\r\n";
await ssl.WriteAsync(System.Text.Encoding.ASCII.GetBytes(request), cts.Token);
await ssl.FlushAsync(cts.Token);
var buffer = new byte[4096];
var read = await ssl.ReadAsync(buffer, cts.Token);
Report("ttfb", sw.Elapsed - t0, System.Text.Encoding.ASCII.GetString(buffer, 0, Math.Min(read, 40)).Split('\n')[0].Trim());
// 5. Drain the body
t0 = sw.Elapsed;
long total = read;
while ((read = await ssl.ReadAsync(buffer, cts.Token)) > 0) total += read;
Report("body", sw.Elapsed - t0, $"{total} bytes");
Report("total", sw.Elapsed);
}
catch (OperationCanceledException)
{
Console.WriteLine($"TIMEOUT after {sw.Elapsed.TotalSeconds:F1}s: the phase printed last is the one BEFORE the stall.");
return 2;
}
catch (Exception ex)
{
Console.WriteLine($"FAILED after {sw.Elapsed.TotalMilliseconds:F0} ms: {ex.GetType().Name}: {ex.Message}");
return 1;
}
return 0;Run it with dotnet run -- api.screening-partner.example 443 /v1/health. The project needs only the default console template targeting net10.0.
A client configured for a NAT gateway
using System.Net.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
public static class PartnerClientRegistration
{
public static IServiceCollection AddScreeningClient(this IServiceCollection services)
{
services.AddHttpClient("screening", c =>
{
c.BaseAddress = new Uri("https://api.screening-partner.example/");
c.Timeout = Timeout.InfiniteTimeSpan; // the resilience handler owns timeouts
})
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
ConnectTimeout = TimeSpan.FromSeconds(3), // fail fast on blocked paths
PooledConnectionLifetime = TimeSpan.FromMinutes(2), // re-resolve DNS regularly
PooledConnectionIdleTimeout = TimeSpan.FromSeconds(240), // under the NAT 350 s idle timeout
KeepAlivePingDelay = TimeSpan.FromSeconds(60), // used for HTTP/2 connections
KeepAlivePingTimeout = TimeSpan.FromSeconds(10)
})
.AddStandardResilienceHandler(o =>
{
o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(8);
o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
o.Retry.MaxRetryAttempts = 2; // ONE retry layer, not one per component
o.Retry.UseJitter = true;
o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30); // at least twice the attempt timeout
});
return services;
}
}Code explained line by line
- The probe prints each phase on its own line. When a call stalls, the last printed phase tells you the stall happened in the next one: a printed
dnsand noconnectmeans the connect hung, which points to the network path or partner allow-list. CancellationTokenSource(TimeSpan.FromSeconds(20))is a hard deadline for the whole probe so the tool itself cannot hang.Dns.GetHostAddressesAsyncshows every address returned. If the partner has several IPs and only one is blocked, this and the connect line reveal it.AuthenticateAsClientAsyncwithTargetHostsets SNI, and the certificate line prints the negotiated protocol and the expiry in days: a one-glance certificate health check.- The raw HTTP request avoids
HttpClienton purpose, so you measure the network and TLS without any pooling, retry or proxy behaviour of your application. ConnectTimeoutof 3 seconds turns a blackholed path into a quick failure instead of waiting for the operating system's much longer default.PooledConnectionLifetimeforces connections to be replaced regularly, so a changed DNS answer is picked up. Without it, a long-livedHttpClientcan keep using an old IP for days.PooledConnectionIdleTimeoutbelow 350 seconds means your client closes idle connections before the NAT gateway silently resets them, which removes a class of "first call after quiet period fails" errors.c.Timeout = Timeout.InfiniteTimeSpanhands timeout control to the resilience handler (attempt timeout 2 s inside, total 8 s outside), which is the pattern in the partner API lesson.- One retry layer only: the AWS SDK clients have their own retries (see Retry strategies), so do not stack a third layer in a queue consumer without counting the multiplication.
Security considerations
⚠ Learning shortcut — not for production
The probe uses the default certificate validation, which is correct. During an incident people are tempted to "just ignore the TLS error" (ServerCertificateCustomValidationCallback = (_, _, _, _) => true) to see whether the call works. Never ship or even deploy that. It disables the protection that stops a man-in-the-middle from reading payment data. Use the probe's certificate details to prove the cause, then fix the certificate or trust store properly.
- Fail closed for controls. A screening or fraud timeout must result in a held payment and a retry later, never an automatic pass. Write this in the design document and test it.
- Keep the NAT allow-list under change control. Replacing a NAT gateway can break a partner's firewall rule. Treat Elastic IPs as part of the partner contract and reuse them, or notify the partner before a change.
- mTLS client certificates expire. Alarm on expiry well before the date, and keep the certificate in Secrets Manager, not in the image.
- Do not paste credentials or full payloads into partner tickets. Send timestamps, correlation IDs, IPs and the timing breakdown.
- Egress control stays on. While diagnosing, do not widen outbound security group or Network Firewall rules to "allow all". Add the specific destination and record the change. See Segmentation and egress.
- Nothing here substitutes for your third-party risk process; see Operational resilience and third-party risk.
Banking use case
For Harbourline, the screening call sits in the payment path with a business rule: "no screening result, no release". The Payments Service writes the payment as screening, calls the partner with an 8-second total budget and, on timeout or an open circuit, leaves the payment in screening and puts a retry message on an SQS queue with a delay. The customer sees "Payment pending". A dashboard shows partner latency by phase, the circuit state and the number of payments held. In the 14:05 incident, the phase timings showed DNS and connect at normal speed but time-to-first-byte of 25 seconds, so the team could tell the partner with evidence, reduced concurrency to avoid piling on, and kept payments safe while the partner fixed an overloaded database on their side.
Common mistakes
- No connect timeout. Blackholed paths wait for the operating system default (which can be minutes), exhausting threads.
- A new
HttpClientper request. Causes port exhaustion on the NAT gateway and repeated TLS handshakes. UseIHttpClientFactoryor a singleton. - Retry at every layer. SDK retries, resilience-handler retries and queue redelivery multiply. Decide who owns retries.
- Retrying non-idempotent calls without a key. A retried payment submission is the duplicate payment playbook again.
- Testing from a laptop. The laptop has a different path and IP. Always test from the workload's subnet.
- Ignoring
Retry-Afteron 429. Retrying immediately makes you a bad neighbour and gets you blocked for longer. - Failing open. Treating a timeout as "no match" in a screening flow.
- Forgetting the NAT IP is a contract. A new gateway means a new IP, which the partner must allow.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Connect hangs, Flow Logs show REJECT | Security group or NACL blocks outbound or return traffic | Add the specific rule (NACLs are stateless: allow ephemeral ports back) |
| Connect hangs, no flow records at all | No route to the NAT gateway, or the gateway is in a failed AZ | Check the route table and NAT gateway state, use a NAT gateway per AZ |
| Works from one AZ, times out from another | Different NAT gateway (and a different, non-allow-listed IP) per AZ | Ask the partner to allow every NAT IP, or use shared addresses by design |
ErrorPortAllocation greater than zero | Too many connections to one destination per NAT address | Reuse connections, add Elastic IPs, reduce parallelism |
| First call after a quiet period fails, retry succeeds | NAT idle timeout reset a pooled connection | Lower pooled idle timeout under 350 seconds, enable keep-alives |
AuthenticationException in the TLS phase | Partner or client certificate expired, wrong chain, or wrong host name | Inspect with the probe or openssl s_client, rotate, fix the trust store |
| Calls fail only for a day after a partner change | Stale DNS in long-lived connections | Set PooledConnectionLifetime, restart tasks if urgent |
| HTTP 429 | Partner rate limit reached | Honour Retry-After, reduce concurrency, ask for a higher limit |
| Timeouts rise when load rises | Our thread pool or connection pool is starved, not the partner | Check pool metrics and request queue time, cap concurrency with a bulkhead |
| Only some payments time out, large ones | Large body or slow client upload through a constrained path | Compare body size, check the transfer phase |
Cost and cleanup
Cost label: free. The probe runs anywhere. The optional sandbox exercise (a Fargate task in a private subnet behind a NAT gateway) is paid: a NAT gateway costs roughly a few cents per hour plus data processing (⚠ VERIFY current price), so do it only as deploy-and-destroy within an hour, or use the free design-only variant below.
# If you created a sandbox NAT gateway and Elastic IP for this exercise
aws ec2 delete-nat-gateway --nat-gateway-id <nat-id> --profile learn-sandbox
# wait until the state is "deleted", then release the address
aws ec2 release-address --allocation-id <eipalloc-id> --profile learn-sandbox
# Remove the alarm if you created it
aws cloudwatch delete-alarms --alarm-names hl-nat-port-allocation-errors --profile learn-sandbox
# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-partner --profile learn-sandboxExercises
- 🟢 Guided: Run the probe against
example.comon port 443 from your machine. Read all five lines. Then run it against a port that is not listening on a host you control (for example a local port) and describe which line is the last one printed and why. - 🟡 Challenge: Extend the probe to run ten times and print the p50 and maximum for each phase. Then add a
--resolveoption that forces a specific IP, so you can test each address the DNS answer returned. - 🔴 Live problem: "Calls to the partner started timing out at 14:05 from Availability Zone B only. Zone A is fine. The partner says they see nothing." You have Flow Logs, NAT gateway metrics and CloudTrail. List your first five checks in order, and say what you will send the partner.
Assignment
Create a partner dependency runbook page for your portfolio: a table of phases, the evidence for each, the fail-safe behaviour for one partner, the NAT IPs the partner must allow, the certificate expiry dates and the contact path. Commit it as runbooks/partner-api-timeouts.md with fictional data.
Solutions
Against example.com you see dns, connect, tls, ttfb and body lines, then total. Against a closed local port, connect fails immediately with SocketException: Connection refused, and the output stops at the dns line, because the connect was the failing phase. A refused connection means a host is reachable but nothing listens; a timeout with no response (for example a firewall dropping packets) would show the TIMEOUT message after 20 seconds with dns as the last line. Why: distinguishing refused from silent timeout is the most useful single signal, because refused means "reachable" and silence means "blocked or dropped".
Loop the phase measurements into lists, then sort and take the median (middle element) and the last. A --resolve host:port:ip option skips Dns.GetHostAddressesAsync and uses the supplied IPAddress for connect, while still passing the host name as TargetHost for SNI. Testing each address from the DNS answer reveals the partner node that is slow or blocked. Why: partner platforms often have several nodes, and a single bad one shows up as intermittent errors that look random.
One AZ failing points to something per AZ, which is the NAT gateway and its Elastic IP. Checks: (1) route table of the AZ B private subnet: does it point to the NAT gateway in AZ B and is that gateway available? (2) NAT gateway B metrics: ErrorPortAllocation, PacketsDropCount; (3) Flow Logs for tasks in AZ B toward the partner IP: REJECT or no flows? (4) CloudTrail for recent CreateNatGateway, ReplaceRoute or ReleaseAddress events around 14:00 that would have changed the source IP; (5) compare the Elastic IPs of the two gateways with the partner's allow-list. To the partner send: the UTC window, the source IP(s) of the failing AZ, the destination hostname and IP, the probe timings (DNS and connect failing or slow) and two correlation IDs, and ask whether that source IP is on their allow-list and whether they see connection attempts from it. Why: a per-AZ pattern strongly suggests a per-AZ egress address, and the allow-list is the most common cause.
Knowledge check
Answer: The TCP connect. Look at the network path: outbound security group, NACL (stateless), route to the NAT gateway, NAT health, and whether the partner's firewall allows your NAT IP. Flow Logs REJECT entries help separate the cases.
Answer: Releasing without a screening result would make the control optional whenever the partner is slow. The safe default is to fail closed: hold, retry later and release only on a real answer.
Answer: The NAT gateway resets connections idle for more than 350 seconds without telling the client. A pooled connection reused after that fails on first use. Closing idle connections earlier avoids it.
Answer: Up to 3 x 3 x 3 = 27 attempts in the worst case if layers multiply. Decide which layer owns retries and keep the others at zero or minimal.
Answer: The laptop uses a different network path, DNS resolver, source IP and firewall rules, so it cannot reproduce NAT, security group, NACL or allow-list problems.
Interview questions
- A partner API times out intermittently. How do you investigate? Time DNS, connect, TLS, first byte and transfer separately from the same subnet, check Flow Logs and NAT metrics, compare AZs and source IPs, then send the partner evidence.
- How do you design a call to an unreliable partner? Connect, attempt and total timeouts, bounded jittered retries at one layer only, a circuit breaker, a bulkhead, idempotency keys for state-changing calls and an explicit fail-safe behaviour.
- What is the danger of a retry storm? Retries multiply across layers and clients and overload the recovering partner, delaying recovery. Backoff, jitter, circuit breakers and honouring
Retry-Afterlimit it. - How do you handle a partner's rate limit? Respect 429 and
Retry-After, cap concurrency, queue work and agree limits in the contract. - Why does a NAT gateway matter to a partner integration? Its Elastic IP is the identity the partner allows, and it has connection and idle-timeout limits that affect how you pool connections.
Certification practice
Workloads in a private subnet call a partner API through a NAT gateway. Calls succeed for a few minutes after deployment, then intermittently fail with connection resets. CloudWatch shows ErrorPortAllocation greater than zero. What is the most likely fix?
A. Replace the NAT gateway with a larger instance type.
B. Reuse HTTP connections (a shared HttpClient) and, if needed, add Elastic IPs to the NAT gateway.
C. Remove the network ACL.
D. Increase the security group's inbound rules.
Answer: B. Port allocation errors mean too many concurrent connections to one destination per address. Reusing connections reduces them, and extra addresses add capacity. NAT gateways are not sized by instance type (A), NACLs are not the cause (C), and inbound security group rules do not affect outbound NAT ports (D).
A Lambda function calls a partner API and sometimes hits its 10-second timeout. The SDK-based client has no explicit timeout and retries three times. Which change best reduces blocked invocations?
A. Raise the function timeout to 15 minutes. B. Set explicit attempt and total timeouts that fit inside the function timeout, with a limited number of retries. C. Remove all retries and timeouts. D. Move the function out of the VPC.
Answer: B. Timeouts should be shorter than the caller's deadline so the function can fail cleanly and respond. A hides the problem and raises cost, C risks indefinite hangs, and D is not related to the stated cause.
A company must let only its private-subnet applications reach a partner's API, and the partner requires a fixed source IP address. Which design meets the requirement?
A. Give every instance a public IP address. B. Route private subnets through a NAT gateway with an Elastic IP, and give the partner that IP to allow. C. Use a VPC peering connection to the partner's public IP. D. Use an internet gateway attached to the private subnets with no NAT.
Answer: B. A NAT gateway keeps instances private and presents a stable Elastic IP. A exposes instances, C does not apply to public internet endpoints, and D does not work for private addresses without NAT.
Completion checklist
- I can name the five phases of an outbound call and what each failure looks like
- I can list the hops from a private subnet to a partner and one blocker for each
- I know why a screening timeout must fail closed
- I ran the probe and read the output
- I can explain the NAT idle timeout and port exhaustion limits and the client settings that avoid them
- I ran the cleanup steps and confirmed nothing tagged
m17-playbook-partnerremains
Key takeaways
- Time each phase (DNS, connect, TLS, first byte, transfer) from inside the failing subnet.
- Per-AZ or per-IP patterns point to the NAT gateway and the partner's allow-list.
- Own retries in one layer, with attempt and total timeouts, backoff with jitter and a circuit breaker.
- Decide the fail-safe behaviour before the incident: for controls, fail closed.
- Never disable TLS validation to "test"; use the evidence to fix the real cause.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Playbook: end-of-day batch overrunning its window
Find the slow stage of a nightly batch, tell data skew from saturation from waiting, buy time without skipping controls, and add progress-based alarms so the next overrun is caught at 01:00 instead of 05:00.
Playbook: AccessDenied and IAM policy evaluation
Turn an AccessDenied error into a located cause in minutes: read the message, confirm who is really calling, find the denying layer with CloudTrail, the policy simulator and Access Analyzer, and fix it with the smallest safe change.