Skip to Content
Cloud Engineering Academy
4. API Development & Integration

Calling partner APIs safely: egress, secrets, timeouts, retries and circuit breakers

Call external partners such as a sanctions-screening or credit-reference API from AWS with controlled egress, cached secrets, outbound mTLS and a tested Microsoft.Extensions.Http.Resilience pipeline that respects partner rate limits.

intermediate⏱ 2 h 🟢 FreeDVA-C03SAA-C03SCS-C03

Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 (Polly v8) · AWSSDK.SecretsManager.Caching 3.0.0 · AWS SDK for .NET v4 · Region us-east-1. Verified against official sources on 2026-11-17.

Learning objectives

By the end of this lesson you can:

  • Design egress control for partner calls: private subnets, a NAT gateway with a fixed Elastic IP, and a domain allow-list with AWS Network Firewall or an egress proxy.
  • Load partner credentials from AWS Secrets Manager with in-process caching, and present a client certificate for outbound mTLS.
  • Build a typed HttpClient with Microsoft.Extensions.Http.Resilience: per-attempt and total timeouts, retries with exponential backoff and jitter, a circuit breaker, a concurrency limiter, and Retry-After handling.
  • Decide when not to retry (non-idempotent calls) and when hedging is appropriate.
  • Prove the behaviour with a fault-injecting fake partner and xUnit tests (Lab 4.4).
  • Explain the AWS SDK's own standard and adaptive retry modes and why you do not wrap SDK calls in Polly retries.

Prerequisites

  • Webhooks — the inbound side of partner integration
  • Idempotency keys — retries are only safe for idempotent operations
  • Module 2: Networking — subnets, route tables, NAT gateways
  • Tools: .NET 10 SDK (or Docker). No AWS resources are needed for the lab.

Why this matters in banking

Every outbound Harbourline payment is screened by a (fictional) sanctions partner, Sentinel Screening, before it is released. On a busy morning Sentinel's latency climbs from 200 ms to 20 s. Harbourline's Payments Service has the default HttpClient timeout of 100 seconds and three naive retries per call. Within minutes every thread is waiting on Sentinel, the Payments Service stops answering health checks, the load balancer drains it, and all payments stop — including ones that did not need screening yet. The partner had a slowdown; Harbourline turned it into an outage. Meanwhile Sentinel's firewall starts rejecting calls because a redeployment moved the service to a subnet without the allow-listed IP.

Partner calls fail in ordinary ways — slowness, 503s, 429s, expired certificates, changed IPs. The skill is to fail fast, bounded and visibly, and never to amplify the partner's problem.

The idea in plain language

Calling a partner safely has two halves:

  1. The network path. Your code runs in private subnets with no direct internet access. Outbound traffic leaves through a NAT gateway with a fixed public IP (an Elastic IP) that the partner allow-lists. Optionally, a firewall or proxy checks that traffic goes only to approved domains.
  2. The client behaviour. Each call has a per-attempt timeout and an overall deadline; only transient failures on idempotent requests are retried, with growing, randomised delays; a circuit breaker stops calling a partner that is clearly down; a concurrency limiter (bulkhead) caps how many calls can be in flight; and a 429 with Retry-After is honoured.

Analogy: Phoning a correspondent bank. You call from the switchboard number they have on file (the Elastic IP), only to numbers on the approved list (domain allow-list). If nobody answers in ten seconds you hang up (attempt timeout), try again a bit later (backoff), and give up after a minute (total deadline). If the line has been dead all morning, the switchboard stops putting calls through for a while (circuit breaker). Only five operators may be on hold to that bank at once (bulkhead). And if they say "call back in two minutes", you do (Retry-After).

What problem the AWS services solve

ConcernAWS optionSharp edges
Fixed source IP for partner allow-listsNAT gateway (zonal) with an Elastic IP, in a public subnetAbout USD 0.045 per hour and USD 0.045 per GB processed, plus USD 0.005 per hour per public IPv4 address (pricing page examples, checked 2026-11-17; confirm us-east-1 in the calculator ⚠ VERIFY) — roughly USD 33 per month per gateway plus about USD 3.65 for its IPv4 address, before data
HA egress without a NAT per AZRegional NAT gateway (--availability-mode regional)No public subnet needed; expands across AZs automatically (up to 60 minutes for a new AZ); billed USD 0.045 per hour per AZ it spans; one EIP per AZ, so the partner must allow-list several IPs; no private NAT
Domain allow-list for egressAWS Network Firewall stateful domain list (TLS_SNI, HTTP_HOST, optionally AWS_ROUTE_53_RESOLVED_DOMAIN)About USD 0.395 per endpoint-hour and USD 0.065 per GB (pricing page examples); NAT gateway hourly and data charges are waived for traffic that also passes the firewall. SNI can be spoofed unless you add resolved-domain matching
Cheaper allow-listEgress proxy (for example Squid on Fargate)You run, patch and scale it
Partner credentialsAWS Secrets Manager + AWSSDK.SecretsManager.CachingAbout USD 0.40 per secret per month plus USD 0.05 per 10,000 calls; cache in process
AWS API calls without NATInterface VPC endpoints (Secrets Manager, STS, ...)About USD 0.01 per endpoint-ENI hour plus USD 0.01 per GB

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure NAT Gateway with a public IPNAT gateway + Elastic IP (zonal or regional)Zonal NAT lives in a public subnet per AZ; regional NAT needs none
Azure Firewall FQDN rulesNetwork Firewall domain list rule groupsDomain rules see only TLS SNI / HTTP Host unless you add resolved-domain matching
Key Vault + Azure.Extensions.AspNetCore.Configuration.SecretsSecrets Manager + caching client, or Amazon.Extensions.Configuration.SystemsManagerSame patterns: cache in memory, reload on rotation
Polly in IHttpClientFactoryIdentical: Microsoft.Extensions.Http.Resilience (built on Polly v8)Nothing AWS-specific — your skills transfer directly
Azure SDK retry options (RetryOptions)AWS SDK RetryMode (Standard / Adaptive) and MaxErrorRetryv4 defaults to Standard mode
Client certificate from Key Vault for mTLSPFX stored in Secrets Manager (or an exportable ACM certificate ⚠ VERIFY)You load the certificate into SocketsHttpHandler yourself

Architecture

Inside the service, the resilience pipeline wraps every call (outermost first):

Security-boundary view: the only path to the internet is through the firewall and NAT; the service holds no long-lived secrets in configuration (only secret ARNs); AWS API calls stay on interface endpoints; and the partner sees one stable, auditable source identity (IP + client certificate + API key).

Step-by-step implementation

Fix the egress path

Put workloads in private subnets. Route 0.0.0.0/0 to a NAT gateway (zonal: one per AZ for resilience; or one regional NAT gateway). Give the partner your Elastic IPs and ask for theirs, so both sides can allow-list. Add Network Firewall or a proxy if policy requires a domain allow-list.

Store and cache credentials

Create one secret per partner credential (hl-m04/partners/sentinel/api-key, .../client-cert). Grant the task or function role secretsmanager:GetSecretValue on those ARNs only. Use the caching client so a call costs a memory lookup, not a network round trip; pick a cache TTL shorter than your rotation overlap.

Build a typed client with one resilience handler

AddHttpClient<T>() gives you IHttpClientFactory connection management. Add one resilience handler — the standard one with tuned options, or a custom pipeline. Microsoft's guidance is not to stack several. Set HttpClient.Timeout to infinite so the pipeline owns timing.

Decide what may be retried

Retry only transient failures (5xx, 408, 429, connection errors, attempt timeouts) on idempotent requests. GET is idempotent; a screening POST is safe only because Harbourline sends an Idempotency-Key the partner honours. For other POSTs call options.Retry.DisableForUnsafeHttpMethods().

Inject faults and test

Wrap the real handler chain with a DelegatingHandler that returns 503s, 429s or hangs on a script, and assert attempts, timings and circuit state. This is Lab 4.4.

AWS Console instructions

Design-only by default (a NAT gateway bills hourly). To look without creating anything: VPC console → NAT gateways → Create NAT gateway shows Availability mode: Zonal | Regional; zonal asks for a public subnet and an Elastic IP allocation. Network Firewall → Rule groups → Create rule group → Stateful → Domain list shows the allow/deny choice and the target types. Secrets Manager → Store a new secret for the API key, with the four mandatory tags.

AWS CLI

# Design reference only — these create billable resources.
# Allocate an Elastic IP for the partner allow-list
aws ec2 allocate-address --domain vpc \
  --tag-specifications 'ResourceType=elastic-ip,Tags=[{Key=Project,Value=cea-lab},{Key=Lab,Value=m04-partner},{Key=Environment,Value=learn},{Key=Owner,Value=your-name}]' \
  --profile learn-sandbox --region us-east-1

# Zonal NAT gateway in a public subnet using that allocation
aws ec2 create-nat-gateway --subnet-id subnet-0123456789abcdef0 --allocation-id eipalloc-0123456789abcdef0 \
  --profile learn-sandbox --region us-east-1

# Alternative: regional NAT gateway (no subnet; manual mode lets you attach your own EIPs per AZ)
aws ec2 create-nat-gateway --vpc-id vpc-0123456789abcdef0 --availability-mode regional \
  --profile learn-sandbox --region us-east-1

# Store the partner API key (free-tier credits apply; about USD 0.40/month prorated)
aws secretsmanager create-secret --name hl-m04/partners/sentinel/api-key --secret-string "REPLACE_ME" \
  --tags Key=Project,Value=cea-lab Key=Lab,Value=m04-partner Key=Environment,Value=learn Key=Owner,Value=your-name \
  --profile learn-sandbox --region us-east-1

Infrastructure as Code

using Amazon.CDK;
using Amazon.CDK.AWS.EC2;
using Constructs;

namespace Harbourline.Partners.Infra;

public sealed class PartnerEgressStack : Stack
{
    public PartnerEgressStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        // One Elastic IP we can give to the partner. Two AZs => two EIPs in production.
        var eip = new CfnEIP(this, "PartnerEgressIp", new CfnEIPProps { Domain = "vpc" });

        var vpc = new Vpc(this, "PaymentsVpc", new VpcProps
        {
            VpcName = "hl-m04-payments",
            MaxAzs = 2,
            NatGateways = 1, // learning/cost shortcut: production uses one per AZ or a regional NAT gateway
            NatGatewayProvider = NatProvider.Gateway(new NatGatewayProps
            {
                EipAllocationIds = new[] { eip.AttrAllocationId },
            }),
            SubnetConfiguration = new ISubnetConfiguration[]
            {
                new SubnetConfiguration { Name = "public", SubnetType = SubnetType.PUBLIC, CidrMask = 24 },
                new SubnetConfiguration { Name = "app", SubnetType = SubnetType.PRIVATE_WITH_EGRESS, CidrMask = 22 },
            },
        });

        vpc.AddInterfaceEndpoint("SecretsManager", new InterfaceVpcEndpointOptions
        {
            Service = InterfaceVpcEndpointAwsService.SECRETS_MANAGER,
            PrivateDnsEnabled = true,
        });

        new CfnOutput(this, "PartnerAllowListIp", new CfnOutputProps { Value = eip.Ref });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m04-partner");
        Amazon.CDK.Tags.Of(this).Add("Environment", "learn");
        Amazon.CDK.Tags.Of(this).Add("Owner", "your-name");
    }
}

Complete code

The module project uses this exact pattern in labs/m04-account-information-api/src/Harbourline.AccountInformation.Api/Accounts/CoreBankingAccountStore.cs: a typed HttpClient to the simulated core banking API wrapped with AddResilienceHandler (timeout, retry with jitter, circuit breaker). Below is the sanctions-screening equivalent, plus Lab 4.4's fault injection and tests.

src/Harbourline.Partners/Sanctions/SanctionsScreeningClient.cs
using System.Net.Http.Json;

namespace Harbourline.Partners.Sanctions;

public sealed record ScreeningRequest(string PaymentId, string CreditorName, string CreditorIban, decimal Amount, string Currency);
public sealed record ScreeningResult(string ScreeningId, string Outcome); // "clear" or "potential_match"

public sealed class SanctionsScreeningClient(HttpClient http)
{
    public async Task<ScreeningResult> ScreenAsync(ScreeningRequest request, CancellationToken ct)
    {
        using var message = new HttpRequestMessage(HttpMethod.Post, "v2/screenings")
        {
            Content = JsonContent.Create(request),
        };
        // The partner de-duplicates on this key, which is what makes retrying a POST safe.
        message.Headers.Add("Idempotency-Key", request.PaymentId);

        using var response = await http.SendAsync(message, ct);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadFromJsonAsync<ScreeningResult>(ct)
            ?? throw new InvalidOperationException("Empty screening response.");
    }
}
src/Harbourline.Partners/Resilience/PartnerResilience.cs
using Microsoft.Extensions.Http.Resilience;
using Polly;

namespace Harbourline.Partners.Resilience;

public sealed record PartnerResilienceTimings(TimeSpan Total, TimeSpan Attempt, TimeSpan RetryBaseDelay, TimeSpan BreakDuration)
{
    public static readonly PartnerResilienceTimings Production =
        new(TimeSpan.FromSeconds(8), TimeSpan.FromSeconds(2), TimeSpan.FromMilliseconds(200), TimeSpan.FromSeconds(15));
}

public static class PartnerResilience
{
    public static void Configure(ResiliencePipelineBuilder<HttpResponseMessage> builder, PartnerResilienceTimings t)
    {
        builder
            .AddConcurrencyLimiter(permitLimit: 20, queueLimit: 10)       // bulkhead
            .AddTimeout(t.Total)                                          // overall deadline, including retries
            .AddRetry(new HttpRetryStrategyOptions
            {
                MaxRetryAttempts = 2,
                BackoffType = DelayBackoffType.Exponential,
                UseJitter = true,
                Delay = t.RetryBaseDelay,
                ShouldRetryAfterHeader = true,                            // honour 429/503 Retry-After
            })
            .AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
            {
                FailureRatio = 0.5,
                MinimumThroughput = 4,
                SamplingDuration = TimeSpan.FromSeconds(30),
                BreakDuration = t.BreakDuration,
            })
            .AddTimeout(t.Attempt);                                       // per-attempt timeout
    }
}
src/Harbourline.Partners/Program.cs (registration excerpt)
using System.Net.Security;
using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;
using Amazon.SecretsManager;
using Amazon.SecretsManager.Extensions.Caching;
using Harbourline.Partners.Resilience;
using Harbourline.Partners.Sanctions;

var builder = WebApplication.CreateBuilder(args);

// One cache for the whole process; 5-minute TTL so rotated secrets are picked up quickly.
var secretsClient = new AmazonSecretsManagerClient();
var secrets = new SecretsManagerCache(secretsClient, new SecretCacheConfiguration { CacheItemTTL = 300_000 });
builder.Services.AddSingleton(secrets);

// Client certificate for outbound mTLS, stored as a base64 PFX in Secrets Manager.
var pfxBase64 = await secrets.GetSecretString(builder.Configuration["Partners:Sentinel:ClientCertSecretId"]!);
var clientCert = X509CertificateLoader.LoadPkcs12(Convert.FromBase64String(pfxBase64), password: null);

builder.Services.AddTransient<PartnerApiKeyHandler>();

var sentinel = builder.Services.AddHttpClient<SanctionsScreeningClient>(c =>
{
    c.BaseAddress = new Uri(builder.Configuration["Partners:Sentinel:BaseUrl"]!);
    c.Timeout = Timeout.InfiniteTimeSpan; // the resilience pipeline owns all timeouts
});
sentinel.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
    ConnectTimeout = TimeSpan.FromSeconds(3),
    PooledConnectionLifetime = TimeSpan.FromMinutes(5), // re-resolve DNS if the partner moves
    SslOptions = new SslClientAuthenticationOptions
    {
        ClientCertificates = new X509CertificateCollection { clientCert },
        EnabledSslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13,
    },
});
sentinel.AddResilienceHandler("sentinel", b => PartnerResilience.Configure(b, PartnerResilienceTimings.Production));
sentinel.AddHttpMessageHandler<PartnerApiKeyHandler>(); // inside the pipeline: runs on every attempt

var app = builder.Build();
app.Run();

public sealed class PartnerApiKeyHandler(SecretsManagerCache secrets, IConfiguration config) : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
    {
        var apiKey = await secrets.GetSecretString(config["Partners:Sentinel:ApiKeySecretId"]!);
        request.Headers.Remove("X-Api-Key");
        request.Headers.Add("X-Api-Key", apiKey);
        return await base.SendAsync(request, ct);
    }
}

If you prefer the standard handler, the equivalent configuration is:

Standard handler alternative
sentinel.AddStandardResilienceHandler(o =>
{
    o.RateLimiter.DefaultRateLimiterOptions.PermitLimit = 20;
    o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(8);
    o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
    o.Retry.MaxRetryAttempts = 2;
    o.Retry.Delay = TimeSpan.FromMilliseconds(200);
    o.CircuitBreaker.FailureRatio = 0.5;
    o.CircuitBreaker.MinimumThroughput = 4;
    o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
    o.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(15);
    // For clients whose POSTs are NOT idempotent: o.Retry.DisableForUnsafeHttpMethods();
});

Lab 4.4 — fault-injecting fake partner and tests

tests/Harbourline.Partners.Tests/FakePartner.cs
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Net;
using System.Net.Http.Headers;
using System.Text;

namespace Harbourline.Partners.Tests;

public enum PartnerFault { None, ServiceUnavailable, TooManyRequests, Hang }

/// Sits inside the resilience pipeline and misbehaves according to a script.
public sealed class FaultInjectingHandler(params PartnerFault[] script) : DelegatingHandler
{
    private readonly ConcurrentQueue<PartnerFault> _script = new(script);
    private int _calls;
    public int Calls => _calls;

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
    {
        Interlocked.Increment(ref _calls);
        var fault = _script.TryDequeue(out var next) ? next : PartnerFault.None;
        switch (fault)
        {
            case PartnerFault.ServiceUnavailable:
                return new HttpResponseMessage(HttpStatusCode.ServiceUnavailable);
            case PartnerFault.TooManyRequests:
                var throttled = new HttpResponseMessage(HttpStatusCode.TooManyRequests);
                throttled.Headers.RetryAfter = new RetryConditionHeaderValue(TimeSpan.FromSeconds(1));
                return throttled;
            case PartnerFault.Hang:
                await Task.Delay(Timeout.InfiniteTimeSpan, ct); // only the attempt timeout ends this
                throw new UnreachableException();
            default:
                return await base.SendAsync(request, ct);
        }
    }
}

/// The "partner" itself: always answers 200 with a clear result.
public sealed class FakePartnerHandler : HttpMessageHandler
{
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct) =>
        Task.FromResult(new HttpResponseMessage(HttpStatusCode.OK)
        {
            Content = new StringContent("""{"screeningId":"scr-0001","outcome":"clear"}""", Encoding.UTF8, "application/json"),
        });
}
tests/Harbourline.Partners.Tests/SanctionsClientResilienceTests.cs
using Harbourline.Partners.Resilience;
using Harbourline.Partners.Sanctions;
using Microsoft.Extensions.DependencyInjection;
using Polly.CircuitBreaker;
using Xunit;

namespace Harbourline.Partners.Tests;

public sealed class SanctionsClientResilienceTests
{
    private static readonly PartnerResilienceTimings Fast =
        new(TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(200), TimeSpan.FromMilliseconds(10), TimeSpan.FromSeconds(30));

    private static readonly ScreeningRequest Payment =
        new("PAY-0193a7c2-6b1e-7f00-8000-000000000001", "Northwind Traders Ltd", "GB33BUKB20201555555555", 1250.00m, "GBP");

    private static (SanctionsScreeningClient Client, FaultInjectingHandler Faults) Build(params PartnerFault[] script)
    {
        var faults = new FaultInjectingHandler(script);
        var services = new ServiceCollection();
        var http = services.AddHttpClient<SanctionsScreeningClient>(c => c.BaseAddress = new Uri("https://sentinel.example.net/"));
        http.AddResilienceHandler("test", b => PartnerResilience.Configure(b, Fast));
        http.AddHttpMessageHandler(() => faults);
        http.ConfigurePrimaryHttpMessageHandler(() => new FakePartnerHandler());
        return (services.BuildServiceProvider().GetRequiredService<SanctionsScreeningClient>(), faults);
    }

    [Fact]
    public async Task Retries_transient_503_then_succeeds()
    {
        var (client, faults) = Build(PartnerFault.ServiceUnavailable, PartnerFault.ServiceUnavailable);
        var result = await client.ScreenAsync(Payment, TestContext.Current.CancellationToken);
        Assert.Equal("clear", result.Outcome);
        Assert.Equal(3, faults.Calls);
    }

    [Fact]
    public async Task Attempt_timeout_cuts_a_hanging_call_and_retries()
    {
        var (client, faults) = Build(PartnerFault.Hang);
        var result = await client.ScreenAsync(Payment, TestContext.Current.CancellationToken);
        Assert.Equal("clear", result.Outcome);
        Assert.Equal(2, faults.Calls);
    }

    [Fact]
    public async Task Honours_retry_after_on_429()
    {
        var (client, _) = Build(PartnerFault.TooManyRequests);
        var started = TimeProvider.System.GetTimestamp();
        await client.ScreenAsync(Payment, TestContext.Current.CancellationToken);
        Assert.True(TimeProvider.System.GetElapsedTime(started) >= TimeSpan.FromMilliseconds(900));
    }

    [Fact]
    public async Task Circuit_opens_after_repeated_failures()
    {
        var script = Enumerable.Repeat(PartnerFault.ServiceUnavailable, 10).ToArray();
        var (client, faults) = Build(script);

        await Assert.ThrowsAsync<HttpRequestException>(() => client.ScreenAsync(Payment, TestContext.Current.CancellationToken));
        await Assert.ThrowsAsync<BrokenCircuitException>(() => client.ScreenAsync(Payment, TestContext.Current.CancellationToken));
        Assert.True(faults.Calls <= 4, "an open circuit must stop calls reaching the partner");
    }
}

Code explained line by line

  1. Idempotency-Key = PaymentId — the partner stores the first result per key, so a retried POST cannot create a second screening. Without this header, the retry policy for this client would have to exclude POST.
  2. AddConcurrencyLimiter(20, 10) — the bulkhead: at most 20 calls in flight and 10 queued; the rest fail immediately with RateLimiterRejectedException instead of piling up threads.
  3. Outer AddTimeout(t.Total) — the total deadline across all attempts and back-off delays, so a caller never waits more than 8 seconds.
  4. AddRetry — Exponential + UseJitter spreads retries out so many instances do not hammer the partner in lockstep. ShouldRetryAfterHeader (default true) uses the partner's Retry-After delay when it is sent.
  5. AddCircuitBreaker — opens when at least half of at least 4 calls in a 30-second window fail; while open, calls fail fast with BrokenCircuitException for BreakDuration.
  6. Inner AddTimeout(t.Attempt) — the per-attempt timeout. It is inside retry and circuit breaker, so a slow attempt counts as a failure and is retried.
  7. c.Timeout = Timeout.InfiniteTimeSpan — otherwise HttpClient's own 100-second timeout fights your pipeline.
  8. SecretsManagerCache with CacheItemTTL = 300_000 — milliseconds; the default is one hour. A short TTL lets rotated secrets take effect quickly.
  9. X509CertificateLoader.LoadPkcs12 — the .NET 9+ API for loading a PFX; the certificate's private key never touches disk.
  10. SslClientAuthenticationOptions.ClientCertificates — presents the client certificate during the TLS handshake (outbound mTLS).
  11. PooledConnectionLifetime — recycles connections so a partner DNS change is picked up; also re-establishes TLS periodically.
  12. AddResilienceHandler then AddHttpMessageHandler<PartnerApiKeyHandler>() — handlers added later sit inside; the key handler therefore runs on every attempt and always uses the current cached key.
  13. FaultInjectingHandler — added after the resilience handler, so it sits between the pipeline and the fake partner: exactly where a real network fault would occur.
  14. PartnerFault.Hang — waits on the cancellation token; only the attempt timeout can end it, proving the timeout works.
  15. TestContext.Current.CancellationToken — xUnit v3's per-test token, cancelled if the test run is aborted.
  16. Circuit_opens_after_repeated_failures — the first call fails after three attempts; the fourth recorded failure opens the circuit, so the second call is rejected without reaching the partner.

AWS SDK calls have their own retries

The AWS SDK for .NET v4 already retries AWS calls: RetryMode defaults to Standard (v4 removed Legacy), with MaxErrorRetry 2 for most services (more for DynamoDB). Adaptive mode, still marked experimental, adds client-side rate limiting when AWS throttles you. Configure it per client (new AmazonSecretsManagerConfig { RetryMode = RequestRetryMode.Adaptive, MaxErrorRetry = 3 }) or with AWS_RETRY_MODE / AWS_MAX_ATTEMPTS. Use a CancellationToken for deadlines on async calls — the Timeout property does not affect them. Do not wrap SDK calls in a Polly retry as well: three SDK attempts inside three Polly attempts is nine requests during an AWS throttling event. Details: Calling AWS APIs from C#.

Hedging — rarely, and only for reads

Hedging sends a second request if the first is slow and uses whichever answers first (AddStandardHedgingHandler, default up to 10 attempts with a 2-second delay). It cuts tail latency for idempotent reads against independent endpoints — for example a credit-reference lookup with two regional endpoints. Never hedge non-idempotent calls, and remember it can double your partner traffic and your per-call fees.

Security considerations

⚠ Learning shortcut — not for production

The lab talks to an in-memory fake partner with no TLS, no mTLS and an API key from configuration. In production: TLS 1.2+ with certificate validation left on (never a ServerCertificateCustomValidationCallback that returns true), a client certificate from Secrets Manager or your PKI, API keys only in Secrets Manager, and least-privilege secretsmanager:GetSecretValue on the specific secret ARNs.

⚠ Learning shortcut — not for production

The CDK example uses a single NAT gateway for two AZs to save cost. An AZ failure then cuts all partner traffic. Production uses one zonal NAT gateway per AZ (and tells the partner every Elastic IP) or a regional NAT gateway.

  • Data minimisation. Send the partner only what screening needs (name, IBAN, amount, country). Sharing customer data with a processor needs a data-processing agreement and UK GDPR review — not an engineering decision.
  • Log safely. Log partner, status, latency, attempt number and correlation id; never request bodies with personal data or the API key.
  • Fail closed for compliance checks. If screening is unavailable (circuit open), a payment must wait in a screening state, not be released unscreened.
  • Egress allow-lists protect against data exfiltration as much as against SSRF: a compromised container cannot send data to an arbitrary host.

Banking use case

Harbourline's Payments Service screens payment PAY-... from ACC-0000123456 to "Northwind Traders Ltd" for GBP 1,250.00. The call leaves through the NAT gateway's Elastic IP, presents Harbourline's client certificate and API key, and returns clear in 180 ms. When Sentinel degrades, the circuit opens within seconds; payments queue in screening, a CloudWatch alarm on the breaker's state pages the on-call engineer, and the backlog drains automatically when the circuit half-opens and succeeds.

Common mistakes

  • Only one timeout → either users wait forever or retries never happen. Use attempt and total timeouts.
  • Retrying non-idempotent POSTs → duplicate payments or screenings. Use idempotency keys or disable retries for unsafe methods.
  • Retries without jitter → synchronised retry storms. Keep UseJitter = true.
  • Stacking resilience handlers → multiplied retries. One handler per client.
  • new HttpClient() per call → socket exhaustion and stale DNS. Use IHttpClientFactory.
  • Polly retries around AWS SDK calls → multiplied retries during throttling. Tune the SDK instead.
  • Ignoring 429 → the partner blocks you. Honour Retry-After and add a bulkhead.

Troubleshooting

SymptomLikely causeFix
Partner returns 403 after a deploymentTraffic left through a different IP (new subnet, new NAT)Check route tables; give the partner every NAT Elastic IP
TaskCanceledException after exactly 100 sHttpClient.Timeout default still activeSet Timeout.InfiniteTimeSpan; rely on pipeline timeouts
Options validation error at startup (standard handler)Total timeout not greater than the attempt timeout, or circuit-breaker sampling duration less than double the attempt timeoutFix the values; the built-in validators enforce both rules
BrokenCircuitException in logsPartner failing; circuit openCheck partner status; alarm on it; do not "fix" by removing the breaker
RateLimiterRejectedExceptionBulkhead fullRaise limits only after checking the partner's rate limit; add queueing upstream
TLS handshake fails with mTLSExpired or wrong client certificate, missing intermediateCheck expiry and chain; alarm 30 days before expiry
Calls blocked by Network FirewallDomain missing from allow-list, or client sends no SNIAdd the domain (leading . for subdomains); check firewall alert logs

Cost and cleanup

The lab is free: it runs locally with fakes. The egress architecture is design-only here; a zonal NAT gateway costs roughly USD 33 per month plus USD 0.045 per GB, and Network Firewall roughly USD 288 per endpoint per month (estimates from the pricing pages, checked 2026-11-17). If you create anything to explore, delete it the same day:

aws ec2 delete-nat-gateway --nat-gateway-id nat-0123456789abcdef0 --profile learn-sandbox --region us-east-1
# Wait until the NAT gateway is deleted, then release the address
aws ec2 release-address --allocation-id eipalloc-0123456789abcdef0 --profile learn-sandbox --region us-east-1
aws secretsmanager delete-secret --secret-id hl-m04/partners/sentinel/api-key --force-delete-without-recovery \
  --profile learn-sandbox --region us-east-1
# If you used CDK: cdk destroy --profile learn-sandbox

aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab \
  --profile learn-sandbox --region us-east-1

Exercises

  • 🟢 Guided: Add a test proving that a POST without an Idempotency-Key is not retried when you configure DisableForUnsafeHttpMethods() on a second client.
  • 🟡 Challenge: Add a CrediViewClient for a credit-reference GET with AddStandardHedgingHandler across two fake endpoints, and test that a slow primary is beaten by the secondary.
  • 🔴 Live problem: "Since the partner's maintenance window, about 1% of screenings time out after exactly 2 seconds, then succeed on retry. Partner says p99 latency is 1.8 s. Ops want to raise the attempt timeout to 30 s." Respond with a recommendation.

Assignment

Write a partner integration runbook for Sentinel Screening: egress IPs and how to change them, secrets and rotation steps, timeout/retry/breaker settings with reasons, what happens to payments when the circuit is open, the alarms, and the fault-injection test results from Lab 4.4.

Solutions

Knowledge check

Interview questions

  • How do you stop a slow dependency taking down your service? Attempt and total timeouts, a bulkhead, a circuit breaker, and degrading gracefully (queue work rather than block threads).
  • When is it safe to retry a POST? When the operation is idempotent by design, typically through an idempotency key the server honours.
  • How would you give a partner a stable source IP from Fargate? Private subnets routed through NAT gateways with Elastic IPs (or a regional NAT gateway); give them every IP.
  • What does Retry-After change? The client waits at least the server's requested delay instead of its own back-off; Microsoft.Extensions.Http.Resilience does this by default.
  • Polly directly or Microsoft.Extensions.Http.Resilience? The latter for HttpClient: it integrates with IHttpClientFactory, options, telemetry and request cloning; use Polly v8 pipelines directly for non-HTTP work.

Certification practice

Completion checklist

  • I can explain the egress path from a private subnet to a partner and why the Elastic IP matters
  • I know when a regional NAT gateway is a better fit than zonal ones
  • My typed client has attempt and total timeouts, jittered retries, a circuit breaker and a bulkhead
  • I ran the Lab 4.4 fault-injection tests
  • I can explain why SDK calls are not wrapped in Polly retries
  • I deleted anything I created and confirmed nothing tagged Project=cea-lab remains

Key takeaways

  • Fix the network identity (private subnets, NAT gateway, Elastic IPs, optional domain allow-list) before writing client code.
  • One resilience handler per client: bulkhead, total timeout, jittered retry honouring Retry-After, circuit breaker, attempt timeout.
  • Retry only transient failures on idempotent requests; idempotency keys make POSTs safe.
  • Test resilience with fault injection, not hope.
  • Let the AWS SDK handle AWS retries; do not nest them.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page