Calling partner APIs safely: egress, secrets, timeouts, retries and circuit breakers
Call external partners such as a sanctions-screening or credit-reference API from AWS with controlled egress, cached secrets, outbound mTLS and a tested Microsoft.Extensions.Http.Resilience pipeline that respects partner rate limits.
Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 (Polly v8) · AWSSDK.SecretsManager.Caching 3.0.0 · AWS SDK for .NET v4 · Region us-east-1. Verified against official sources on 2026-11-17.
Learning objectives
By the end of this lesson you can:
- Design egress control for partner calls: private subnets, a NAT gateway with a fixed Elastic IP, and a domain allow-list with AWS Network Firewall or an egress proxy.
- Load partner credentials from AWS Secrets Manager with in-process caching, and present a client certificate for outbound mTLS.
- Build a typed
HttpClientwithMicrosoft.Extensions.Http.Resilience: per-attempt and total timeouts, retries with exponential backoff and jitter, a circuit breaker, a concurrency limiter, andRetry-Afterhandling. - Decide when not to retry (non-idempotent calls) and when hedging is appropriate.
- Prove the behaviour with a fault-injecting fake partner and xUnit tests (Lab 4.4).
- Explain the AWS SDK's own standard and adaptive retry modes and why you do not wrap SDK calls in Polly retries.
Prerequisites
- Webhooks — the inbound side of partner integration
- Idempotency keys — retries are only safe for idempotent operations
- Module 2: Networking — subnets, route tables, NAT gateways
- Tools: .NET 10 SDK (or Docker). No AWS resources are needed for the lab.
Why this matters in banking
Every outbound Harbourline payment is screened by a (fictional) sanctions partner, Sentinel Screening, before it is released. On a busy morning Sentinel's latency climbs from 200 ms to 20 s. Harbourline's Payments Service has the default HttpClient timeout of 100 seconds and three naive retries per call. Within minutes every thread is waiting on Sentinel, the Payments Service stops answering health checks, the load balancer drains it, and all payments stop — including ones that did not need screening yet. The partner had a slowdown; Harbourline turned it into an outage. Meanwhile Sentinel's firewall starts rejecting calls because a redeployment moved the service to a subnet without the allow-listed IP.
Partner calls fail in ordinary ways — slowness, 503s, 429s, expired certificates, changed IPs. The skill is to fail fast, bounded and visibly, and never to amplify the partner's problem.
The idea in plain language
Calling a partner safely has two halves:
- The network path. Your code runs in private subnets with no direct internet access. Outbound traffic leaves through a NAT gateway with a fixed public IP (an Elastic IP) that the partner allow-lists. Optionally, a firewall or proxy checks that traffic goes only to approved domains.
- The client behaviour. Each call has a per-attempt timeout and an overall deadline; only transient failures on idempotent requests are retried, with growing, randomised delays; a circuit breaker stops calling a partner that is clearly down; a concurrency limiter (bulkhead) caps how many calls can be in flight; and a
429withRetry-Afteris honoured.
Analogy: Phoning a correspondent bank. You call from the switchboard number they have on file (the Elastic IP), only to numbers on the approved list (domain allow-list). If nobody answers in ten seconds you hang up (attempt timeout), try again a bit later (backoff), and give up after a minute (total deadline). If the line has been dead all morning, the switchboard stops putting calls through for a while (circuit breaker). Only five operators may be on hold to that bank at once (bulkhead). And if they say "call back in two minutes", you do (
Retry-After).
What problem the AWS services solve
| Concern | AWS option | Sharp edges |
|---|---|---|
| Fixed source IP for partner allow-lists | NAT gateway (zonal) with an Elastic IP, in a public subnet | About USD 0.045 per hour and USD 0.045 per GB processed, plus USD 0.005 per hour per public IPv4 address (pricing page examples, checked 2026-11-17; confirm us-east-1 in the calculator ⚠ VERIFY) — roughly USD 33 per month per gateway plus about USD 3.65 for its IPv4 address, before data |
| HA egress without a NAT per AZ | Regional NAT gateway (--availability-mode regional) | No public subnet needed; expands across AZs automatically (up to 60 minutes for a new AZ); billed USD 0.045 per hour per AZ it spans; one EIP per AZ, so the partner must allow-list several IPs; no private NAT |
| Domain allow-list for egress | AWS Network Firewall stateful domain list (TLS_SNI, HTTP_HOST, optionally AWS_ROUTE_53_RESOLVED_DOMAIN) | About USD 0.395 per endpoint-hour and USD 0.065 per GB (pricing page examples); NAT gateway hourly and data charges are waived for traffic that also passes the firewall. SNI can be spoofed unless you add resolved-domain matching |
| Cheaper allow-list | Egress proxy (for example Squid on Fargate) | You run, patch and scale it |
| Partner credentials | AWS Secrets Manager + AWSSDK.SecretsManager.Caching | About USD 0.40 per secret per month plus USD 0.05 per 10,000 calls; cache in process |
| AWS API calls without NAT | Interface VPC endpoints (Secrets Manager, STS, ...) | About USD 0.01 per endpoint-ENI hour plus USD 0.01 per GB |
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
| Azure NAT Gateway with a public IP | NAT gateway + Elastic IP (zonal or regional) | Zonal NAT lives in a public subnet per AZ; regional NAT needs none |
| Azure Firewall FQDN rules | Network Firewall domain list rule groups | Domain rules see only TLS SNI / HTTP Host unless you add resolved-domain matching |
Key Vault + Azure.Extensions.AspNetCore.Configuration.Secrets | Secrets Manager + caching client, or Amazon.Extensions.Configuration.SystemsManager | Same patterns: cache in memory, reload on rotation |
Polly in IHttpClientFactory | Identical: Microsoft.Extensions.Http.Resilience (built on Polly v8) | Nothing AWS-specific — your skills transfer directly |
Azure SDK retry options (RetryOptions) | AWS SDK RetryMode (Standard / Adaptive) and MaxErrorRetry | v4 defaults to Standard mode |
| Client certificate from Key Vault for mTLS | PFX stored in Secrets Manager (or an exportable ACM certificate ⚠ VERIFY) | You load the certificate into SocketsHttpHandler yourself |
Architecture
Inside the service, the resilience pipeline wraps every call (outermost first):
Security-boundary view: the only path to the internet is through the firewall and NAT; the service holds no long-lived secrets in configuration (only secret ARNs); AWS API calls stay on interface endpoints; and the partner sees one stable, auditable source identity (IP + client certificate + API key).
Step-by-step implementation
Fix the egress path
Put workloads in private subnets. Route 0.0.0.0/0 to a NAT gateway (zonal: one per AZ for resilience; or one regional NAT gateway). Give the partner your Elastic IPs and ask for theirs, so both sides can allow-list. Add Network Firewall or a proxy if policy requires a domain allow-list.
Store and cache credentials
Create one secret per partner credential (hl-m04/partners/sentinel/api-key, .../client-cert). Grant the task or function role secretsmanager:GetSecretValue on those ARNs only. Use the caching client so a call costs a memory lookup, not a network round trip; pick a cache TTL shorter than your rotation overlap.
Build a typed client with one resilience handler
AddHttpClient<T>() gives you IHttpClientFactory connection management. Add one resilience handler — the standard one with tuned options, or a custom pipeline. Microsoft's guidance is not to stack several. Set HttpClient.Timeout to infinite so the pipeline owns timing.
Decide what may be retried
Retry only transient failures (5xx, 408, 429, connection errors, attempt timeouts) on idempotent requests. GET is idempotent; a screening POST is safe only because Harbourline sends an Idempotency-Key the partner honours. For other POSTs call options.Retry.DisableForUnsafeHttpMethods().
Inject faults and test
Wrap the real handler chain with a DelegatingHandler that returns 503s, 429s or hangs on a script, and assert attempts, timings and circuit state. This is Lab 4.4.
AWS Console instructions
Design-only by default (a NAT gateway bills hourly). To look without creating anything: VPC console → NAT gateways → Create NAT gateway shows Availability mode: Zonal | Regional; zonal asks for a public subnet and an Elastic IP allocation. Network Firewall → Rule groups → Create rule group → Stateful → Domain list shows the allow/deny choice and the target types. Secrets Manager → Store a new secret for the API key, with the four mandatory tags.
AWS CLI
# Design reference only — these create billable resources.
# Allocate an Elastic IP for the partner allow-list
aws ec2 allocate-address --domain vpc \
--tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m04-partner},{Key=Environment,Value=learn},{Key=Owner,Value=your-name}]' \
--profile learn-sandbox --region us-east-1
# Zonal NAT gateway in a public subnet using that allocation
aws ec2 create-nat-gateway --subnet-id subnet-0123456789abcdef0 --allocation-id eipalloc-0123456789abcdef0 \
--profile learn-sandbox --region us-east-1
# Alternative: regional NAT gateway (no subnet; manual mode lets you attach your own EIPs per AZ)
aws ec2 create-nat-gateway --vpc-id vpc-0123456789abcdef0 --availability-mode regional \
--profile learn-sandbox --region us-east-1
# Store the partner API key (free-tier credits apply; about USD 0.40/month prorated)
aws secretsmanager create-secret --name hl-m04/partners/sentinel/api-key --secret-string "REPLACE_ME" \
--tags Key=Project,Value=cea-lab Key=Lab,Value=m04-partner Key=Environment,Value=learn Key=Owner,Value=your-name \
--profile learn-sandbox --region us-east-1Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Constructs;
namespace Harbourline.Partners.Infra;
public sealed class PartnerEgressStack : Stack
{
public PartnerEgressStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
// One Elastic IP we can give to the partner. Two AZs => two EIPs in production.
var eip = new CfnEIP(this, "PartnerEgressIp", new CfnEIPProps { Domain = "vpc" });
var vpc = new Vpc(this, "PaymentsVpc", new VpcProps
{
VpcName = "hl-m04-payments",
MaxAzs = 2,
NatGateways = 1, // learning/cost shortcut: production uses one per AZ or a regional NAT gateway
NatGatewayProvider = NatProvider.Gateway(new NatGatewayProps
{
EipAllocationIds = new[] { eip.AttrAllocationId },
}),
SubnetConfiguration = new ISubnetConfiguration[]
{
new SubnetConfiguration { Name = "public", SubnetType = SubnetType.PUBLIC, CidrMask = 24 },
new SubnetConfiguration { Name = "app", SubnetType = SubnetType.PRIVATE_WITH_EGRESS, CidrMask = 22 },
},
});
vpc.AddInterfaceEndpoint("SecretsManager", new InterfaceVpcEndpointOptions
{
Service = InterfaceVpcEndpointAwsService.SECRETS_MANAGER,
PrivateDnsEnabled = true,
});
new CfnOutput(this, "PartnerAllowListIp", new CfnOutputProps { Value = eip.Ref });
Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
Amazon.CDK.Tags.Of(this).Add("Lab", "m04-partner");
Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
}
}Complete code
The module project uses this exact pattern in labs/m04-account-information-api/src/Harbourline.AccountInformation.Api/Accounts/CoreBankingAccountStore.cs: a typed HttpClient to the simulated core banking API wrapped with AddResilienceHandler (timeout, retry with jitter, circuit breaker). Below is the sanctions-screening equivalent, plus Lab 4.4's fault injection and tests.
using System.Net.Http.Json;
namespace Harbourline.Partners.Sanctions;
public sealed record ScreeningRequest(string PaymentId, string CreditorName, string CreditorIban, decimal Amount, string Currency);
public sealed record ScreeningResult(string ScreeningId, string Outcome); // "clear" or "potential_match"
public sealed class SanctionsScreeningClient(HttpClient http)
{
public async Task<ScreeningResult> ScreenAsync(ScreeningRequest request, CancellationToken ct)
{
using var message = new HttpRequestMessage(HttpMethod.Post, "v2/screenings")
{
Content = JsonContent.Create(request),
};
// The partner de-duplicates on this key, which is what makes retrying a POST safe.
message.Headers.Add("Idempotency-Key", request.PaymentId);
using var response = await http.SendAsync(message, ct);
response.EnsureSuccessStatusCode();
return await response.Content.ReadFromJsonAsync<ScreeningResult>(ct)
?? throw new InvalidOperationException("Empty screening response.");
}
}using Microsoft.Extensions.Http.Resilience;
using Polly;
namespace Harbourline.Partners.Resilience;
public sealed record PartnerResilienceTimings(TimeSpan Total, TimeSpan Attempt, TimeSpan RetryBaseDelay, TimeSpan BreakDuration)
{
public static readonly PartnerResilienceTimings Production =
new(TimeSpan.FromSeconds(8), TimeSpan.FromSeconds(2), TimeSpan.FromMilliseconds(200), TimeSpan.FromSeconds(15));
}
public static class PartnerResilience
{
public static void Configure(ResiliencePipelineBuilder<HttpResponseMessage> builder, PartnerResilienceTimings t)
{
builder
.AddConcurrencyLimiter(permitLimit: 20, queueLimit: 10) // bulkhead
.AddTimeout(t.Total) // overall deadline, including retries
.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 2,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
Delay = t.RetryBaseDelay,
ShouldRetryAfterHeader = true, // honour 429/503 Retry-After
})
.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
MinimumThroughput = 4,
SamplingDuration = TimeSpan.FromSeconds(30),
BreakDuration = t.BreakDuration,
})
.AddTimeout(t.Attempt); // per-attempt timeout
}
}using System.Net.Security;
using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Extensions.Caching;
using Harbourline.Partners.Resilience;
using Harbourline.Partners.Sanctions;
var builder = WebApplication.CreateBuilder(args);
// One cache for the whole process; 5-minute TTL so rotated secrets are picked up quickly.
var secretsClient = new AmazonSecretsManagerClient();
var secrets = new SecretsManagerCache(secretsClient, new SecretCacheConfiguration { CacheItemTTL = 300_000 });
builder.Services.AddSingleton(secrets);
// Client certificate for outbound mTLS, stored as a base64 PFX in Secrets Manager.
var pfxBase64 = await secrets.GetSecretString(builder.Configuration["Partners:Sentinel:ClientCertSecretId"]!);
var clientCert = X509CertificateLoader.LoadPkcs12(Convert.FromBase64String(pfxBase64), password: null);
builder.Services.AddTransient<PartnerApiKeyHandler>();
var sentinel = builder.Services.AddHttpClient<SanctionsScreeningClient>(c =>
{
c.BaseAddress = new Uri(builder.Configuration["Partners:Sentinel:BaseUrl"]!);
c.Timeout = Timeout.InfiniteTimeSpan; // the resilience pipeline owns all timeouts
});
sentinel.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
ConnectTimeout = TimeSpan.FromSeconds(3),
PooledConnectionLifetime = TimeSpan.FromMinutes(5), // re-resolve DNS if the partner moves
SslOptions = new SslClientAuthenticationOptions
{
ClientCertificates = new X509CertificateCollection { clientCert },
EnabledSslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13,
},
});
sentinel.AddResilienceHandler("sentinel", b => PartnerResilience.Configure(b, PartnerResilienceTimings.Production));
sentinel.AddHttpMessageHandler<PartnerApiKeyHandler>(); // inside the pipeline: runs on every attempt
var app = builder.Build();
app.Run();
public sealed class PartnerApiKeyHandler(SecretsManagerCache secrets, IConfiguration config) : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
var apiKey = await secrets.GetSecretString(config["Partners:Sentinel:ApiKeySecretId"]!);
request.Headers.Remove("X-Api-Key");
request.Headers.Add("X-Api-Key", apiKey);
return await base.SendAsync(request, ct);
}
}If you prefer the standard handler, the equivalent configuration is:
sentinel.AddStandardResilienceHandler(o =>
{
o.RateLimiter.DefaultRateLimiterOptions.PermitLimit = 20;
o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(8);
o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
o.Retry.MaxRetryAttempts = 2;
o.Retry.Delay = TimeSpan.FromMilliseconds(200);
o.CircuitBreaker.FailureRatio = 0.5;
o.CircuitBreaker.MinimumThroughput = 4;
o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
o.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(15);
// For clients whose POSTs are NOT idempotent: o.Retry.DisableForUnsafeHttpMethods();
});Lab 4.4 — fault-injecting fake partner and tests
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Net;
using System.Net.Http.Headers;
using System.Text;
namespace Harbourline.Partners.Tests;
public enum PartnerFault { None, ServiceUnavailable, TooManyRequests, Hang }
/// Sits inside the resilience pipeline and misbehaves according to a script.
public sealed class FaultInjectingHandler(params PartnerFault[] script) : DelegatingHandler
{
private readonly ConcurrentQueue<PartnerFault> _script = new(script);
private int _calls;
public int Calls => _calls;
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
Interlocked.Increment(ref _calls);
var fault = _script.TryDequeue(out var next) ? next : PartnerFault.None;
switch (fault)
{
case PartnerFault.ServiceUnavailable:
return new HttpResponseMessage(HttpStatusCode.ServiceUnavailable);
case PartnerFault.TooManyRequests:
var throttled = new HttpResponseMessage(HttpStatusCode.TooManyRequests);
throttled.Headers.RetryAfter = new RetryConditionHeaderValue(TimeSpan.FromSeconds(1));
return throttled;
case PartnerFault.Hang:
await Task.Delay(Timeout.InfiniteTimeSpan, ct); // only the attempt timeout ends this
throw new UnreachableException();
default:
return await base.SendAsync(request, ct);
}
}
}
/// The "partner" itself: always answers 200 with a clear result.
public sealed class FakePartnerHandler : HttpMessageHandler
{
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct) =>
Task.FromResult(new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StringContent("""{"screeningId":"scr-0001","outcome":"clear"}""", Encoding.UTF8, "application/json"),
});
}using Harbourline.Partners.Resilience;
using Harbourline.Partners.Sanctions;
using Microsoft.Extensions.DependencyInjection;
using Polly.CircuitBreaker;
using Xunit;
namespace Harbourline.Partners.Tests;
public sealed class SanctionsClientResilienceTests
{
private static readonly PartnerResilienceTimings Fast =
new(TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(200), TimeSpan.FromMilliseconds(10), TimeSpan.FromSeconds(30));
private static readonly ScreeningRequest Payment =
new("PAY-0193a7c2-6b1e-7f00-8000-000000000001", "Northwind Traders Ltd", "GB33BUKB20201555555555", 1250.00m, "GBP");
private static (SanctionsScreeningClient Client, FaultInjectingHandler Faults) Build(params PartnerFault[] script)
{
var faults = new FaultInjectingHandler(script);
var services = new ServiceCollection();
var http = services.AddHttpClient<SanctionsScreeningClient>(c => c.BaseAddress = new Uri("https://sentinel.example.net/"));
http.AddResilienceHandler("test", b => PartnerResilience.Configure(b, Fast));
http.AddHttpMessageHandler(() => faults);
http.ConfigurePrimaryHttpMessageHandler(() => new FakePartnerHandler());
return (services.BuildServiceProvider().GetRequiredService<SanctionsScreeningClient>(), faults);
}
[Fact]
public async Task Retries_transient_503_then_succeeds()
{
var (client, faults) = Build(PartnerFault.ServiceUnavailable, PartnerFault.ServiceUnavailable);
var result = await client.ScreenAsync(Payment, TestContext.Current.CancellationToken);
Assert.Equal("clear", result.Outcome);
Assert.Equal(3, faults.Calls);
}
[Fact]
public async Task Attempt_timeout_cuts_a_hanging_call_and_retries()
{
var (client, faults) = Build(PartnerFault.Hang);
var result = await client.ScreenAsync(Payment, TestContext.Current.CancellationToken);
Assert.Equal("clear", result.Outcome);
Assert.Equal(2, faults.Calls);
}
[Fact]
public async Task Honours_retry_after_on_429()
{
var (client, _) = Build(PartnerFault.TooManyRequests);
var started = TimeProvider.System.GetTimestamp();
await client.ScreenAsync(Payment, TestContext.Current.CancellationToken);
Assert.True(TimeProvider.System.GetElapsedTime(started) >= TimeSpan.FromMilliseconds(900));
}
[Fact]
public async Task Circuit_opens_after_repeated_failures()
{
var script = Enumerable.Repeat(PartnerFault.ServiceUnavailable, 10).ToArray();
var (client, faults) = Build(script);
await Assert.ThrowsAsync<HttpRequestException>(() => client.ScreenAsync(Payment, TestContext.Current.CancellationToken));
await Assert.ThrowsAsync<BrokenCircuitException>(() => client.ScreenAsync(Payment, TestContext.Current.CancellationToken));
Assert.True(faults.Calls <= 4, "an open circuit must stop calls reaching the partner");
}
}Code explained line by line
Idempotency-Key = PaymentId— the partner stores the first result per key, so a retriedPOSTcannot create a second screening. Without this header, the retry policy for this client would have to excludePOST.AddConcurrencyLimiter(20, 10)— the bulkhead: at most 20 calls in flight and 10 queued; the rest fail immediately withRateLimiterRejectedExceptioninstead of piling up threads.- Outer
AddTimeout(t.Total)— the total deadline across all attempts and back-off delays, so a caller never waits more than 8 seconds. AddRetry—Exponential+UseJitterspreads retries out so many instances do not hammer the partner in lockstep.ShouldRetryAfterHeader(defaulttrue) uses the partner'sRetry-Afterdelay when it is sent.AddCircuitBreaker— opens when at least half of at least 4 calls in a 30-second window fail; while open, calls fail fast withBrokenCircuitExceptionforBreakDuration.- Inner
AddTimeout(t.Attempt)— the per-attempt timeout. It is inside retry and circuit breaker, so a slow attempt counts as a failure and is retried. c.Timeout = Timeout.InfiniteTimeSpan— otherwiseHttpClient's own 100-second timeout fights your pipeline.SecretsManagerCachewithCacheItemTTL = 300_000— milliseconds; the default is one hour. A short TTL lets rotated secrets take effect quickly.X509CertificateLoader.LoadPkcs12— the .NET 9+ API for loading a PFX; the certificate's private key never touches disk.SslClientAuthenticationOptions.ClientCertificates— presents the client certificate during the TLS handshake (outbound mTLS).PooledConnectionLifetime— recycles connections so a partner DNS change is picked up; also re-establishes TLS periodically.AddResilienceHandlerthenAddHttpMessageHandler<PartnerApiKeyHandler>()— handlers added later sit inside; the key handler therefore runs on every attempt and always uses the current cached key.FaultInjectingHandler— added after the resilience handler, so it sits between the pipeline and the fake partner: exactly where a real network fault would occur.PartnerFault.Hang— waits on the cancellation token; only the attempt timeout can end it, proving the timeout works.TestContext.Current.CancellationToken— xUnit v3's per-test token, cancelled if the test run is aborted.Circuit_opens_after_repeated_failures— the first call fails after three attempts; the fourth recorded failure opens the circuit, so the second call is rejected without reaching the partner.
AWS SDK calls have their own retries
The AWS SDK for .NET v4 already retries AWS calls: RetryMode defaults to Standard (v4 removed Legacy), with MaxErrorRetry 2 for most services (more for DynamoDB). Adaptive mode, still marked experimental, adds client-side rate limiting when AWS throttles you. Configure it per client (new AmazonSecretsManagerConfig { RetryMode = RequestRetryMode.Adaptive, MaxErrorRetry = 3 }) or with AWS_RETRY_MODE / AWS_MAX_ATTEMPTS. Use a CancellationToken for deadlines on async calls — the Timeout property does not affect them. Do not wrap SDK calls in a Polly retry as well: three SDK attempts inside three Polly attempts is nine requests during an AWS throttling event. Details: Calling AWS APIs from C#.
Hedging — rarely, and only for reads
Hedging sends a second request if the first is slow and uses whichever answers first (AddStandardHedgingHandler, default up to 10 attempts with a 2-second delay). It cuts tail latency for idempotent reads against independent endpoints — for example a credit-reference lookup with two regional endpoints. Never hedge non-idempotent calls, and remember it can double your partner traffic and your per-call fees.
Security considerations
⚠ Learning shortcut — not for production
The lab talks to an in-memory fake partner with no TLS, no mTLS and an API key from configuration. In production: TLS 1.2+ with certificate validation left on (never a ServerCertificateCustomValidationCallback that returns true), a client certificate from Secrets Manager or your PKI, API keys only in Secrets Manager, and least-privilege secretsmanager:GetSecretValue on the specific secret ARNs.
⚠ Learning shortcut — not for production
The CDK example uses a single NAT gateway for two AZs to save cost. An AZ failure then cuts all partner traffic. Production uses one zonal NAT gateway per AZ (and tells the partner every Elastic IP) or a regional NAT gateway.
- Data minimisation. Send the partner only what screening needs (name, IBAN, amount, country). Sharing customer data with a processor needs a data-processing agreement and UK GDPR review — not an engineering decision.
- Log safely. Log partner, status, latency, attempt number and correlation id; never request bodies with personal data or the API key.
- Fail closed for compliance checks. If screening is unavailable (circuit open), a payment must wait in a
screeningstate, not be released unscreened. - Egress allow-lists protect against data exfiltration as much as against SSRF: a compromised container cannot send data to an arbitrary host.
Banking use case
Harbourline's Payments Service screens payment PAY-... from ACC-0000123456 to "Northwind Traders Ltd" for GBP 1,250.00. The call leaves through the NAT gateway's Elastic IP, presents Harbourline's client certificate and API key, and returns clear in 180 ms. When Sentinel degrades, the circuit opens within seconds; payments queue in screening, a CloudWatch alarm on the breaker's state pages the on-call engineer, and the backlog drains automatically when the circuit half-opens and succeeds.
Common mistakes
- Only one timeout → either users wait forever or retries never happen. Use attempt and total timeouts.
- Retrying non-idempotent POSTs → duplicate payments or screenings. Use idempotency keys or disable retries for unsafe methods.
- Retries without jitter → synchronised retry storms. Keep
UseJitter = true. - Stacking resilience handlers → multiplied retries. One handler per client.
new HttpClient()per call → socket exhaustion and stale DNS. UseIHttpClientFactory.- Polly retries around AWS SDK calls → multiplied retries during throttling. Tune the SDK instead.
- Ignoring 429 → the partner blocks you. Honour
Retry-Afterand add a bulkhead.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Partner returns 403 after a deployment | Traffic left through a different IP (new subnet, new NAT) | Check route tables; give the partner every NAT Elastic IP |
TaskCanceledException after exactly 100 s | HttpClient.Timeout default still active | Set Timeout.InfiniteTimeSpan; rely on pipeline timeouts |
| Options validation error at startup (standard handler) | Total timeout not greater than the attempt timeout, or circuit-breaker sampling duration less than double the attempt timeout | Fix the values; the built-in validators enforce both rules |
BrokenCircuitException in logs | Partner failing; circuit open | Check partner status; alarm on it; do not "fix" by removing the breaker |
RateLimiterRejectedException | Bulkhead full | Raise limits only after checking the partner's rate limit; add queueing upstream |
| TLS handshake fails with mTLS | Expired or wrong client certificate, missing intermediate | Check expiry and chain; alarm 30 days before expiry |
| Calls blocked by Network Firewall | Domain missing from allow-list, or client sends no SNI | Add the domain (leading . for subdomains); check firewall alert logs |
Cost and cleanup
The lab is free: it runs locally with fakes. The egress architecture is design-only here; a zonal NAT gateway costs roughly USD 33 per month plus USD 0.045 per GB, and Network Firewall roughly USD 288 per endpoint per month (estimates from the pricing pages, checked 2026-11-17). If you create anything to explore, delete it the same day:
aws ec2 delete-nat-gateway --nat-gateway-id nat-0123456789abcdef0 --profile learn-sandbox --region us-east-1
# Wait until the NAT gateway is deleted, then release the address
aws ec2 release-address --allocation-id eipalloc-0123456789abcdef0 --profile learn-sandbox --region us-east-1
aws secretsmanager delete-secret --secret-id hl-m04/partners/sentinel/api-key --force-delete-without-recovery \
--profile learn-sandbox --region us-east-1
# If you used CDK: cdk destroy --profile learn-sandbox
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
--profile learn-sandbox --region us-east-1Exercises
- 🟢 Guided: Add a test proving that a
POSTwithout anIdempotency-Keyis not retried when you configureDisableForUnsafeHttpMethods()on a second client. - 🟡 Challenge: Add a
CrediViewClientfor a credit-referenceGETwithAddStandardHedgingHandleracross two fake endpoints, and test that a slow primary is beaten by the secondary. - 🔴 Live problem: "Since the partner's maintenance window, about 1% of screenings time out after exactly 2 seconds, then succeed on retry. Partner says p99 latency is 1.8 s. Ops want to raise the attempt timeout to 30 s." Respond with a recommendation.
Assignment
Write a partner integration runbook for Sentinel Screening: egress IPs and how to change them, secrets and rotation steps, timeout/retry/breaker settings with reasons, what happens to payments when the circuit is open, the alarms, and the fault-injection test results from Lab 4.4.
Solutions
Register a second typed client with AddStandardResilienceHandler(o => o.Retry.DisableForUnsafeHttpMethods()), put FaultInjectingHandler(PartnerFault.ServiceUnavailable) inside it, send a POST, and assert one call and a 503 result. Why: DisableForUnsafeHttpMethods disables retries for POST, PATCH, PUT, DELETE and CONNECT, so only the explicitly idempotent screening client retries POSTs.
Use AddStandardHedgingHandler(r => r.ConfigureOrderedGroups(o => o.Groups.Add(new UriEndpointGroup { Endpoints = { new() { Uri = new("https://a.example.net") }, new() { Uri = new("https://b.example.net") } } }))), set the hedging delay low (for example 100 ms) in options, make the primary fake delay 2 s and the secondary answer immediately, and assert total time is well under 2 s. Why: hedging trades extra load for lower tail latency, which is acceptable only for idempotent reads.
Do not set 30 s. With a p99 of 1.8 s, a 2 s attempt timeout leaves almost no headroom; raise it modestly (for example 3 s) and keep the total deadline (8 s) so a stalled partner still cannot block payments. Ask the partner for p99.9 and check whether timeouts correlate with connection setup (TLS with mTLS after PooledConnectionLifetime expiry). Why: a huge attempt timeout reintroduces the thread-exhaustion outage; the right value sits just above the partner's high percentiles.
Knowledge check
Answer: The attempt timeout limits each individual try and sits inside the retry strategy, so a slow try is retried. The total timeout is outside the retry strategy and caps the whole operation, including back-off delays.
Answer: Without jitter, many clients that failed together retry together, creating synchronised spikes that keep the partner overloaded. Randomised delays spread the load.
Answer: Lambda and Fargate IPs change. Routing through a NAT gateway with an Elastic IP gives a stable, known source address.
Answer: The TLS SNI. It does not decrypt traffic by default, so add AWS_ROUTE_53_RESOLVED_DOMAIN matching or IP rules if SNI spoofing is a concern.
Answer: No. The SDK already retries (Standard mode by default in v4); adding Polly multiplies attempts. Configure RetryMode and MaxErrorRetry instead.
Interview questions
- How do you stop a slow dependency taking down your service? Attempt and total timeouts, a bulkhead, a circuit breaker, and degrading gracefully (queue work rather than block threads).
- When is it safe to retry a POST? When the operation is idempotent by design, typically through an idempotency key the server honours.
- How would you give a partner a stable source IP from Fargate? Private subnets routed through NAT gateways with Elastic IPs (or a regional NAT gateway); give them every IP.
- What does
Retry-Afterchange? The client waits at least the server's requested delay instead of its own back-off;Microsoft.Extensions.Http.Resiliencedoes this by default. - Polly directly or
Microsoft.Extensions.Http.Resilience? The latter forHttpClient: it integrates withIHttpClientFactory, options, telemetry and request cloning; use Polly v8 pipelines directly for non-HTTP work.
Certification practice
An application on ECS Fargate in private subnets across two AZs must call a partner API that only accepts traffic from allow-listed IPs. The solution must survive the loss of one AZ. What should the architect do?
A. Assign public IPs to the Fargate tasks. B. Create a NAT gateway with an Elastic IP in a public subnet in each AZ, route each private subnet to its local NAT gateway, and give the partner both Elastic IPs. C. Create one NAT gateway in one AZ and route both private subnets to it. D. Use a gateway VPC endpoint for the partner API.
Answer: B. Per-AZ NAT gateways keep egress working if one AZ fails, and Elastic IPs are stable. A exposes tasks and IPs change. C is a single-AZ dependency. D: gateway endpoints exist only for S3 and DynamoDB.
A .NET service retries failed POST /transfers calls to a partner using an HTTP resilience handler. The partner occasionally processes a request but the response times out, leading to duplicate transfers. What is the BEST fix?
A. Increase the attempt timeout to 60 seconds. B. Send an idempotency key per transfer that the partner uses to de-duplicate, or disable retries for unsafe HTTP methods. C. Add a second resilience handler with a circuit breaker. D. Switch to the AWS SDK adaptive retry mode.
Answer: B. Retrying non-idempotent requests is safe only with server-side de-duplication. A reduces but cannot eliminate the window. C stacks handlers and does not address duplicates. D applies to AWS SDK calls, not partner HTTP calls.
A bank must ensure workloads in private subnets can reach only three approved partner domains over HTTPS. Which solution meets this with a managed AWS service?
A. Security groups with the partners' domain names as destinations. B. AWS Network Firewall with a stateful domain list rule group in allow-list mode for the three domains, inserted in the route path before the NAT gateway. C. Network ACLs that allow port 443 to 0.0.0.0/0. D. AWS WAF on the NAT gateway.
Answer: B. Network Firewall domain lists filter egress by SNI/Host (and optionally resolved domain). A: security groups take IPs/prefix lists, not domain names. C allows everything on 443. D: WAF does not attach to NAT gateways.
During DynamoDB throttling, logs show each logical write being sent nine times. The code wraps PutItemAsync in a Polly retry with two retries. What explains this and what should change?
A. DynamoDB duplicates writes; enable DynamoDB Streams.
B. The SDK's own retries run inside each Polly attempt; remove the Polly retry and tune the SDK RetryMode/MaxErrorRetry (consider Adaptive mode).
C. The circuit breaker is half-open.
D. The Lambda function has too much memory.
Answer: B. Nested retries multiply attempts. A is false; DynamoDB does not duplicate writes. C and D do not cause repeated sends.
Completion checklist
- I can explain the egress path from a private subnet to a partner and why the Elastic IP matters
- I know when a regional NAT gateway is a better fit than zonal ones
- My typed client has attempt and total timeouts, jittered retries, a circuit breaker and a bulkhead
- I ran the Lab 4.4 fault-injection tests
- I can explain why SDK calls are not wrapped in Polly retries
- I deleted anything I created and confirmed nothing tagged
Project=cea-labremains
Key takeaways
- Fix the network identity (private subnets, NAT gateway, Elastic IPs, optional domain allow-list) before writing client code.
- One resilience handler per client: bulkhead, total timeout, jittered retry honouring
Retry-After, circuit breaker, attempt timeout. - Retry only transient failures on idempotent requests; idempotency keys make POSTs safe.
- Test resilience with fault injection, not hope.
- Let the AWS SDK handle AWS retries; do not nest them.
Further reading
- Build resilient HTTP apps (Microsoft Learn)
- HttpStandardResilienceOptions and HttpRetryStrategyOptions.ShouldRetryAfterHeader
- Polly rate limiter strategy
- Regional NAT gateways and Amazon VPC pricing
- Network Firewall domain list rule groups and pricing
- AWS Secrets Manager caching client for .NET
- Retries and timeouts in the AWS SDK for .NET v4
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Lab: Webhook receiver with HMAC verification and replay protection (Lab 4.3)
Run and extend a .NET 10 webhook receiver that verifies an HMAC-SHA256 signature over the raw body, rejects stale and future timestamps, acknowledges duplicates without processing them twice, supports key rotation, and a sender that retries safely and refuses to call private addresses.
API keys, usage plans, throttling and rate limiting
Use API Gateway throttling, usage plans and API keys for what they are good at — metering and protecting capacity — add per-customer limits in ASP.NET Core, and handle 429s correctly on the client.