Resilience patterns in .NET: timeouts, retries, breakers, bulkheads and load shedding
Make a .NET API survive a slow or failing dependency with a layered resilience pipeline, a retry budget, a circuit breaker, a bulkhead, load shedding and a graceful fallback, and prove each layer with a fault-injecting test backend.
Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 (Polly v8) · AWS SDK for .NET v4 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Classify a failure as transient, slow, permanent or overload, and pick the pattern that fits each.
- Build a layered resilience pipeline for
HttpClientwithMicrosoft.Extensions.Http.Resilience: bulkhead, total timeout, retry with backoff and jitter, circuit breaker and attempt timeout, in the right order. - Explain why retries without a budget cause outages, and add a retry budget.
- Decide which operations may be retried (and why a payment POST must not be).
- Add load shedding with priorities, and a fallback that returns a labelled stale value.
- Configure AWS SDK for .NET v4 retries without multiplying them with your own.
- Test every layer deterministically with a fault-injecting backend, with no AWS account.
Prerequisites
- ELB for resilience — health checks and why dependency failure must be handled inside calls
- AWS SDK for .NET v4 and AWS clients and dependency injection
- Comfortable with
HttpClientFactoryandasync/await - Tools: Docker (tests run in a container), optional .NET 10 SDK
The code for this lesson is labs/m12-resilient-account-api (see Lab: Resilient Account API).
Why this matters in banking
At 10:02 on a Monday, Core Banking begins answering balance requests in 6 seconds instead of 80 milliseconds. Nothing has crashed; every health check passes. Here is what happens in a service with no resilience patterns:
- Each request now holds a thread and a connection for 6 seconds.
- The thread pool and the HTTP connection pool fill up. New requests queue.
- Customer apps time out after 10 seconds and retry, doubling the load.
- The API's own health check, which is unrelated to Core Banking, now times out because there are no free threads. The load balancer removes tasks. The remaining tasks take all the traffic and fail faster.
- A slow dependency has become a complete outage of the API, including endpoints that never use Core Banking.
This is a cascading failure, and slowness is more dangerous than failure, because failures are fast. The patterns in this lesson exist to turn "slow dependency" into "some features degrade, the rest keep working".
The idea in plain language
A resilience pattern is a rule for how your code behaves when a call does not go well. You already know the cousins from .NET: HttpClient.Timeout, CancellationToken, Polly, WCF's OperationTimeout, and ReliableSession. The AWS-specific part is small: the SDK has its own retry settings, and AWS services throttle and fail over in ways you must tolerate.
Six patterns do nearly all the work:
| Pattern | One-sentence idea | Protects against |
|---|---|---|
| Timeout | Stop waiting after a limit | Slow dependencies holding your resources |
| Retry (backoff and jitter) | Try again, with growing and randomised delays | Brief, transient faults |
| Circuit breaker | After repeated failures, stop calling for a while | Hammering a dependency that is down |
| Bulkhead | Limit how much of your capacity one dependency may use | One slow dependency exhausting everything |
| Load shedding | Refuse some work early to keep serving the rest | Overload |
| Fallback | Return something useful (stale, default) instead of failing | Failures you can degrade gracefully |
Analogy: a bank branch handles a broken card reader like this. The cashier gives up on a card after 10 seconds (timeout), tries once more after a pause (retry), and after three failures puts a "reader out of order" sign up and stops trying for 15 minutes (breaker). Only two counters are allowed to deal with card problems so the others keep serving cash (bulkhead). When the queue goes out the door, the manager lets the "pay a bill" customers in and asks the "open a savings account" customers to come back later (load shedding). A customer who only wants their balance is shown the last receipt, clearly dated (fallback).
Classify the failure first
| Failure | Looks like | What helps | What hurts |
|---|---|---|---|
| Transient | A single 503, a dropped connection, a throttle (HTTP 429 or an AWS throttling exception) | Retry with backoff and jitter | Immediate retries; unbounded retries |
| Slow (brown-out) | Responses take seconds, no errors | Timeouts, bulkhead, breaker (slow-call), load shedding | Long timeouts; retrying slow calls |
| Permanent | 404, 400, validation error, a bug | Do not retry. Return the answer | Retrying; masking with a fallback |
| Overload | Many callers, saturated service | Backoff, retry budget, load shedding, rate limiting | Retry storms |
| Outage | Everything fails fast | Circuit breaker, fallback, failover | Retrying every call |
The three rules that keep retries safe
- Only retry what is safe to repeat. A GET is naturally idempotent. A POST that moves money is not. Either do not retry it, or make it idempotent with an idempotency key that the server de-duplicates. The lab pipeline never retries
POST,PUT,PATCHorDELETE. - Add jitter. If 1,000 clients all retry after exactly 200 ms, then 400 ms, they hit the service in synchronised waves. Randomising each delay spreads the load. Exponential backoff plus jitter is the standard.
- Cap retries as a share of traffic. If every request retries twice and the dependency is down, you send three times the load to something already failing. A retry budget says "retries may be at most 10 percent of requests". When the budget is spent, fail fast.
A related hazard is retry amplification across layers. If the browser retries 3 times, the API retries 3 times, and the AWS SDK retries 3 times, one click becomes 27 calls. Retry at one layer, usually the one closest to the user-visible operation, and tune the others down.
What problem the AWS service solves
This lesson is mostly .NET, but the AWS context shapes the design:
- AWS services throttle and fail over. DynamoDB, SQS and others return throttling errors under load, and databases fail over in tens of seconds (Multi-AZ by default). Your code must expect brief errors.
- The AWS SDK for .NET has built-in retries. Per the SDK v4 developer guide, the retry mode is controlled by
RetryMode(Legacy,Standard,Adaptivewhich is documented as experimental) andMaxErrorRetry, which can also come fromAWS_RETRY_MODEandAWS_MAX_ATTEMPTSor the shared config file. If nothing is set, the SDK defaults to Legacy mode with a defaultMaxErrorRetrythat the documentation lists as 4 for Legacy and 2 for Standard ⚠ VERIFY the numbers for your SDK version. Retrying again in Polly on top of this multiplies attempts. - SDK timeouts: for async calls, use the
CancellationTokenparameter to impose a deadline. TheTimeoutproperty on the client config does not affect async calls, per the same guide. - The Polly-based libraries are the Microsoft-supported way to add the other patterns to
HttpClient.
Limitations to be honest about: patterns reduce risk but do not remove it. A breaker can open on a false alarm. A fallback can serve stale data for too long. Bulkhead sizing is a guess until you measure. All of these need monitoring and tuning, and every parameter in the lab is a starting point, not a recommendation.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS / .NET equivalent | What's different |
|---|---|---|
| Polly (v7 policies) | Polly v8 ResiliencePipeline | Pipelines replace policies; strategies are added in order, outermost first |
Microsoft.Extensions.Http.Polly | Microsoft.Extensions.Http.Resilience | The new package; AddStandardResilienceHandler is the one-liner |
Azure SDK built-in retry (RetryOptions) | AWS SDK RetryMode and MaxErrorRetry | Both retry internally; avoid stacking your own on top blindly |
Azure API Management retry, rate-limit policies | API Gateway throttling, WAF rate rules, your own RateLimiter | Edge limits protect you; in-process limiters protect dependencies |
| Azure Front Door/App Gateway circuit breaking (limited) | In-process breakers; Envoy/App Mesh outlier detection if using a mesh | Breakers usually live in the client library |
System.Threading.RateLimiting | Same library, used by Polly's concurrency limiter | The bulkhead is a concurrency limiter |
Polly Fallback policy | Polly FallbackStrategyOptions or plain try/catch | A fallback is a business decision as much as a code pattern |
Architecture
The lab pipeline wraps every call to Core Banking. The first layer added is the outermost.
The circuit breaker has three states:
In the Open state calls fail immediately without touching the dependency, which both protects it (it can recover) and protects you (no threads wasted waiting).
Step-by-step implementation
Add the packages
dotnet add package Microsoft.Extensions.Http.Resilience --version 10.11.0The package brings in Polly v8 and its rate-limiting strategy. For tests, the lab uses a scripted HttpMessageHandler, so no extra mocking library is required.
Start from the standard handler, then own the pipeline
For a first project, AddStandardResilienceHandler() gives you a sensible pipeline in one line. The documented defaults are: rate limiter permit 1,000 with queue 0; total timeout 30 s; retry max 3, exponential, jitter on, base delay 2 s; circuit breaker failure ratio 10%, minimum throughput 100, sampling 30 s, break 5 s; attempt timeout 10 s. Those values suit a generic internet call, not a balance lookup that must answer in 200 ms. For Harbourline we replace them with our own pipeline (AddResilienceHandler) so that every number is a decision.
Order the layers deliberately
Outermost to innermost: bulkhead, total timeout, retry, breaker, attempt timeout. The total timeout must be outside the retry (so it caps all attempts); the attempt timeout must be inside it (so each try is bounded); the breaker sits inside the retry so each attempt is counted, and an open circuit short-circuits quickly. Put the bulkhead outermost so rejected calls cost almost nothing.
Make retries safe
Retry only transient results (connection errors, our own timeouts, 408, 429, 5xx). Never retry unsafe HTTP methods. Add a retry budget. Do not retry an open circuit.
Add load shedding in front
Classify requests (payments and health are critical; reports are best-effort) and refuse low-priority work first when in-flight requests approach capacity. Return 503 with Retry-After.
Decide your fallbacks
For each dependency, write one line: "if this fails we show ___ or we return ___". For balances, the answer is the last known balance labelled with its time. For "is there enough money to pay", the answer is fail (no fallback).
Prove it
Write a test per layer against a fault-injecting backend, then run the demo and read the table. If you cannot make each layer visibly fire, you do not know that it works.
AWS Console instructions
The patterns live in code, but you observe them in the console:
- CloudWatch metrics for your own breaker and retry counters: Console → CloudWatch → Metrics → All metrics → your custom namespace (Module 11 shows how to emit them).
- Throttling you are retrying through: Console → CloudWatch → Metrics → DynamoDB (or SQS) → Table/Queue metrics and look for throttled-request metrics ⚠ VERIFY the exact metric names for your service.
- Service quotas: Console → Service Quotas to see the limits behind throttling errors and request increases before peaks.
AWS CLI
PROFILE=learn-sandbox
# Show the SDK/CLI retry settings in effect for the profile. Unset values mean the defaults.
aws configure get retry_mode --profile $PROFILE
aws configure get max_attempts --profile $PROFILE
# Set retry behaviour for tools that read the shared config (the AWS SDK for .NET reads these too).
aws configure set retry_mode standard --profile $PROFILE
aws configure set max_attempts 3 --profile $PROFILE
# The same through environment variables (apply to the .NET SDK as well):
export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=3max_attempts counts total attempts in the CLI, whereas MaxErrorRetry in the .NET SDK counts retries after the first attempt; keep this difference in mind when comparing numbers ⚠ VERIFY for your versions.
Infrastructure as Code
Resilience patterns are application code, so there is no CDK or Terraform for the patterns themselves. The infrastructure that supports them is in the lab stack: the ReadinessGate and health-check settings, the AZ spread and the scaling policies (Auto Scaling everywhere). One thing you can configure in infrastructure is the SDK retry behaviour for a container, through environment variables:
// On the ECS container definition: make the AWS SDK retry once and let your own pipeline own the rest.
taskDefinition.AddContainer("api", new ContainerDefinitionOptions
{
Image = ContainerImage.FromRegistry(props.ContainerImage),
Environment = new Dictionary<string, string>
{
["AWS_RETRY_MODE"] = "standard",
["AWS_MAX_ATTEMPTS"] = "3", // check how your SDK version counts attempts ⚠ VERIFY
},
});Complete code
The files below are in labs/m12-resilient-account-api/src/Harbourline.Resilience. Together with the tests they run on any machine with Docker.
namespace Harbourline.Resilience;
public sealed class ResilienceOptions
{
public int MaxConcurrentCalls { get; set; } = 50; // bulkhead
public int QueuedCalls { get; set; } // 0 = reject immediately
public TimeSpan TotalTimeout { get; set; } = TimeSpan.FromSeconds(8);
public TimeSpan AttemptTimeout { get; set; } = TimeSpan.FromSeconds(2);
public int MaxRetryAttempts { get; set; } = 2;
public TimeSpan RetryBaseDelay { get; set; } = TimeSpan.FromMilliseconds(200);
public RetryBudget? RetryBudget { get; set; }
public double BreakerFailureRatio { get; set; } = 0.5;
public int BreakerMinimumThroughput { get; set; } = 20;
public TimeSpan BreakerSamplingDuration { get; set; } = TimeSpan.FromSeconds(30);
public TimeSpan BreakerBreakDuration { get; set; } = TimeSpan.FromSeconds(15);
public void Validate()
{
if (MaxConcurrentCalls < 1) throw new ArgumentException("MaxConcurrentCalls must be at least 1.", nameof(MaxConcurrentCalls));
if (MaxRetryAttempts < 0) throw new ArgumentException("MaxRetryAttempts cannot be negative.", nameof(MaxRetryAttempts));
if (AttemptTimeout <= TimeSpan.Zero) throw new ArgumentException("AttemptTimeout must be positive.", nameof(AttemptTimeout));
if (TotalTimeout <= AttemptTimeout) throw new ArgumentException("TotalTimeout must be longer than AttemptTimeout, otherwise retries can never happen.", nameof(TotalTimeout));
if (BreakerFailureRatio is <= 0.0 or > 1.0) throw new ArgumentException("BreakerFailureRatio must be in (0, 1].", nameof(BreakerFailureRatio));
if (BreakerMinimumThroughput < 2) throw new ArgumentException("BreakerMinimumThroughput must be at least 2.", nameof(BreakerMinimumThroughput));
}
}using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Polly;
using Polly.Retry;
using Polly.Timeout;
namespace Harbourline.Resilience;
public static class HarbourlineResilience
{
public const string PipelineName = "harbourline-core-banking";
public static IHttpClientBuilder AddHarbourlineResilience(this IHttpClientBuilder builder, ResilienceOptions options)
{
options.Validate();
builder.AddResilienceHandler(PipelineName, pipeline =>
{
// 1. Bulkhead: cap concurrent calls.
pipeline.AddConcurrencyLimiter(options.MaxConcurrentCalls, options.QueuedCalls);
// 2. Total timeout: one logical call never takes longer than this, retries included.
pipeline.AddTimeout(options.TotalTimeout);
// 3. Retry with exponential backoff + jitter, only for transient faults and safe methods.
if (options.MaxRetryAttempts > 0)
{
var retry = new HttpRetryStrategyOptions
{
MaxRetryAttempts = options.MaxRetryAttempts,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
Delay = options.RetryBaseDelay,
ShouldHandle = args => ValueTask.FromResult(ShouldRetry(args, options.RetryBudget)),
};
retry.DisableForUnsafeHttpMethods(); // must come AFTER ShouldHandle is set
pipeline.AddRetry(retry);
}
// 4. Circuit breaker.
pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = options.BreakerFailureRatio,
MinimumThroughput = options.BreakerMinimumThroughput,
SamplingDuration = options.BreakerSamplingDuration,
BreakDuration = options.BreakerBreakDuration,
});
// 5. Attempt timeout: each try has its own, shorter, limit.
pipeline.AddTimeout(new HttpTimeoutStrategyOptions { Timeout = options.AttemptTimeout });
});
return builder;
}
private static bool ShouldRetry(RetryPredicateArguments<HttpResponseMessage> args, RetryBudget? budget)
{
if (args.AttemptNumber == 0)
{
budget?.RecordRequest(); // each logical request earns a little retry allowance
}
return IsTransient(args.Outcome) && (budget?.TryWithdraw() ?? true);
}
internal static bool IsTransient(Outcome<HttpResponseMessage> outcome)
{
if (outcome.Exception is not null)
{
return outcome.Exception is HttpRequestException or TimeoutRejectedException;
}
return outcome.Result is { } response && (int)response.StatusCode is 408 or 429 or >= 500;
}
}namespace Harbourline.Resilience;
public sealed class RetryBudget
{
private readonly object _gate = new();
private readonly double _retryRatio;
private readonly double _maxTokens;
private double _tokens;
public RetryBudget(double retryRatio = 0.1, double maxTokens = 10)
{
if (retryRatio is <= 0.0 or > 1.0) throw new ArgumentOutOfRangeException(nameof(retryRatio));
if (maxTokens < 1.0) throw new ArgumentOutOfRangeException(nameof(maxTokens));
_retryRatio = retryRatio;
_maxTokens = maxTokens;
_tokens = maxTokens;
}
public void RecordRequest() { lock (_gate) { _tokens = Math.Min(_maxTokens, _tokens + _retryRatio); } }
public bool TryWithdraw()
{
lock (_gate)
{
if (_tokens < 1.0) return false;
_tokens -= 1.0;
return true;
}
}
}public sealed class LoadShedder
{
private readonly int _capacity, _normalLimit, _bestEffortLimit;
private int _inFlight;
public LoadShedder(int capacity, double normalShare = 0.8, double bestEffortShare = 0.5)
{
_capacity = capacity;
_normalLimit = Math.Max(1, (int)Math.Ceiling(capacity * normalShare));
_bestEffortLimit = Math.Max(1, (int)Math.Ceiling(capacity * bestEffortShare));
}
public Lease? TryEnter(RequestPriority priority)
{
var limit = priority switch
{
RequestPriority.Critical => _capacity,
RequestPriority.Normal => _normalLimit,
_ => _bestEffortLimit,
};
if (Interlocked.Increment(ref _inFlight) > limit)
{
Interlocked.Decrement(ref _inFlight);
return null; // shed
}
return new Lease(this);
}
// Lease.Dispose() decrements _inFlight exactly once (full file in the lab).
}The balance read with graceful degradation:
public async Task<BalanceView?> GetAsync(string accountId, CancellationToken cancellationToken = default)
{
try
{
var result = await client.GetBalanceAsync(accountId, cancellationToken);
var view = new BalanceView(result.AccountId, result.Balance, result.Currency, clock.GetUtcNow(), IsStale: false);
_lastKnown[accountId] = view;
return view;
}
catch (Exception ex) when (IsDependencyFailure(ex))
{
return _lastKnown.TryGetValue(accountId, out var last) ? last with { IsStale = true } : null;
}
}
// A timeout, an open circuit, a full bulkhead, a connection error or an HTTP 5xx may be covered by a stale answer.
// A 404 is an ANSWER ("no such account") and is never hidden.
private static bool IsDependencyFailure(Exception ex) => ex switch
{
TimeoutRejectedException or BrokenCircuitException or RateLimiterRejectedException => true,
HttpRequestException { StatusCode: null } => true,
HttpRequestException { StatusCode: { } status } => (int)status >= 500,
_ => false,
};And the fault-injecting backend that makes all of it testable:
public sealed record Fault(HttpStatusCode Status = HttpStatusCode.OK, TimeSpan Delay = default, bool ConnectionReset = false)
{
public static Fault Ok() => new();
public static Fault Error(HttpStatusCode status) => new(status);
public static Fault Slow(TimeSpan delay) => new(Delay: delay);
public static Fault Reset() => new(ConnectionReset: true);
}
// Call 1 gets the first fault, call 2 the second, and so on; the LAST fault repeats forever.
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
var callNumber = Interlocked.Increment(ref _calls);
var fault = _script[Math.Min(callNumber, _script.Length) - 1];
if (fault.Delay > TimeSpan.Zero) await Task.Delay(fault.Delay, ct);
if (fault.ConnectionReset) throw new HttpRequestException("Connection reset by peer (injected fault).");
return new HttpResponseMessage(fault.Status)
{
Content = new StringContent(BalanceJson, Encoding.UTF8, "application/json"),
RequestMessage = request,
};
}A test that proves the circuit breaker fires and recovers:
[Fact]
public async Task The_circuit_opens_after_repeated_failures_and_stops_calling_the_backend()
{
var backend = new ScriptedBackend(Fault.Error(HttpStatusCode.InternalServerError));
var options = FastOptions(); // MinimumThroughput = 4, BreakDuration = 600 ms
options.MaxRetryAttempts = 0; // one backend call per logical call keeps the arithmetic obvious
await using var provider = Build(backend, options);
var client = provider.GetRequiredService<CoreBankingClient>();
for (var i = 0; i < 4; i++)
{
await Assert.ThrowsAsync<HttpRequestException>(() => client.GetBalanceAsync("ACC-0000000001"));
}
Assert.Equal(4, backend.Calls);
// The circuit is now open: calls fail immediately and never reach the backend.
await Assert.ThrowsAsync<BrokenCircuitException>(() => client.GetBalanceAsync("ACC-0000000001"));
Assert.Equal(4, backend.Calls);
}What the demo shows
dotnet run --project src/Harbourline.Resilience.Demo replays four scenarios. This is real output from one run (timings vary):
Scenario Call Outcome Backend calls Time
--------------------------------------------------------------------------------------------------------
Healthy 1 OK 125.40 GBP 1 218 ms
2 OK 125.40 GBP 1 1 ms
3 OK 125.40 GBP 1 0 ms
Two blips, then healthy 1 OK 125.40 GBP 3 169 ms
2 OK 125.40 GBP 1 0 ms
3 OK 125.40 GBP 1 0 ms
Brown-out (every answer takes 3 s) 1 TimeoutRejectedException 3 3231 ms
2 TimeoutRejectedException 2 2117 ms
Hard outage (always HTTP 500) 1 HttpRequestException 3 213 ms
2 HttpRequestException 2 41 ms
3 HttpRequestException 1 4 ms
4 BrokenCircuitException 0 0 ms
5 BrokenCircuitException 0 0 ms
...Code explained line by line
AddResilienceHandler(PipelineName, pipeline => ...)registers a named pipeline on theHttpClient. Strategies are applied in the order added, outermost first.AddConcurrencyLimiter(max, queue)is the bulkhead. With a queue of 0, call numbermax + 1is rejected immediately withRateLimiterRejectedException. Rejecting fast is better than queuing behind a slow dependency.AddTimeout(options.TotalTimeout)placed before the retry caps the whole call. Placed after the retry it would only cap one attempt. The demo's brown-out row shows it: three attempts of 1 s each, about 3.2 s in total, below the 4 s total timeout.HttpRetryStrategyOptionswithBackoffType = ExponentialandUseJitter = trueproduces delays that grow and are randomised.ShouldHandleruns after each attempt. Our predicate records the request onAttemptNumber == 0, then checks the outcome is transient, then asks the budget for permission. In the "hard outage" row the backend calls fall from 3 to 2 to 1 as the budget empties: the budget is visibly doing its job.DisableForUnsafeHttpMethods()wraps the predicate so POST, PUT, PATCH, DELETE and CONNECT are never retried. It must be called afterShouldHandleis assigned, or it wraps the default and is then overwritten.HttpCircuitBreakerStrategyOptionsopens when the failure ratio in the sampling window passes the threshold and at leastMinimumThroughputcalls were seen. In the demo, after six backend calls the circuit opens and calls 4 and later cost zero backend calls and zero milliseconds.HttpTimeoutStrategyOptionsis the attempt timeout, placed innermost so each try is bounded. A cancelled attempt surfaces asTimeoutRejectedException, which is not the standardTimeoutException. The retry predicate and the fallback must handle it explicitly.IsTransienttreats connection errors, our timeouts and HTTP 408, 429 and 5xx as retryable, and deliberately does not retryBrokenCircuitException: retrying an open circuit just adds latency.LoadShedder.TryEnterincrements the in-flight counter first and backs out if over the limit: a lock-free "try acquire". Critical traffic may use all slots; normal 80 percent; best-effort 50 percent.BalanceService.IsDependencyFailurelimits the fallback to genuine dependency failures. A 404 must reach the caller as "no such account".ScriptedBackendis a deterministic fault injector. Because the last fault repeats,new ScriptedBackend(Fault.Error(500))is "always failing" andnew ScriptedBackend(Fault.Error(503), Fault.Reset(), Fault.Ok())is "two blips, then fine".
Security considerations
- Fail closed for security decisions. A fallback may show a stale balance. It must never turn an authorisation failure into "allowed". If the sanctions-screening or fraud service is down, a payment waits or is rejected; it is not approved by default.
- Idempotency is a security and integrity control. Retried or replayed requests must not create two payments. Require
Idempotency-Keyon money-moving POSTs, store results, and return the stored result for a repeat. - Do not log sensitive request bodies in retry telemetry. Log the correlation ID, attempt number and status code, not account numbers or payloads.
- Breakers and shedding are availability controls for you, and attack surface too. An attacker may try to trip a breaker or fill the bulkhead to degrade service for others. Rate limit per client at the edge (WAF, API Gateway) so one caller cannot exhaust shared capacity.
- Stale data must be labelled. Showing a balance without its time can mislead a customer into overspending.
BalanceView.AsOfexists for this reason, and compliance should agree the wording.
⚠ Learning shortcut — not for production
The lab keeps the last known balances in an in-memory dictionary on one process, and its breaker state is per process. Production uses a shared cache (ElastiCache or DynamoDB) with a maximum age, per-dependency pipelines sized from measurements, telemetry exported to CloudWatch or OpenTelemetry, and a documented decision (with the business and compliance) on when stale data may be shown. Also, the demo's timings and thresholds are chosen to finish in seconds, not to be correct for your service.
Banking use case
Harbourline's Accounts API serves a balance in under 200 ms at the 99th percentile on a normal day. The resilience design agreed with the service owner:
| Dependency | Timeout (attempt/total) | Retries | Breaker | Bulkhead | Fallback |
|---|---|---|---|---|---|
| Core Banking balance (GET) | 400 ms / 1.5 s | 1, jittered, budget 10% | 50% over 30 s, min 20 calls, 15 s break | 50 in flight | Last known balance, labelled "as of" |
| Core Banking transfer (POST) | 3 s / 3 s | None; idempotency key and a status lookup | Shared with balance? No, separate pipeline | 20 in flight | None: return 202 with a "pending" status or an error |
| Sanctions screening | 2 s / 5 s | 2 with backoff | 50% over 30 s | 10 in flight | None: hold the payment for manual review |
| Notification service | 1 s / 3 s | 2 | Open fast | 20 in flight | Queue for later (SQS) |
Load shedding: /payments critical, /accounts normal, /reports best-effort. All numbers are fictional examples chosen to show the shape of a decision, not values to copy.
Common mistakes
- Retrying everything. Retrying a 400, a 404 or a payment POST. Retry only transient failures of safe or idempotent operations.
- Retries on every layer. Browser, API, SDK and gateway each retry three times and one request becomes dozens. Pick one layer.
- No jitter. Synchronised retry waves can keep a recovering service down.
- Timeout longer than the caller's timeout. The caller has already given up and retried while you are still working.
- Breaker with no minimum throughput. One failure out of two opens it. Always set a sensible minimum.
- A bulkhead that queues. A long queue behind a slow dependency is a slow outage. Reject fast.
- Catching
TimeoutExceptionand missingTimeoutRejectedException. Polly throws its own type. - Fallbacks that hide permanent errors or approve risky actions. Fall back only for reads where staleness is acceptable.
- Tuning from guesses. Without latency histograms and error rates, every number is folklore.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Retries never happen | TotalTimeout shorter than AttemptTimeout, or retry only wired for a different handler name | Validate() rejects the first; check the pipeline is added to the right HttpClient |
| POSTs are retried | DisableForUnsafeHttpMethods() called before ShouldHandle was set, or a second handler was added | Set ShouldHandle first; use one resilience handler per client |
| Circuit never opens | MinimumThroughput too high for the traffic, or failures not matching the breaker's handled outcomes | Lower the minimum; check the sampling duration; verify 5xx and timeouts are handled |
| Circuit opens on a single failure | MinimumThroughput very low | Raise it (at least 10 to 20 for real traffic) |
BrokenCircuitException forever | Break duration too long or the probe keeps failing | Check the dependency; shorten the break; look at half-open probe results |
RateLimiterRejectedException spikes | Bulkhead too small, or the dependency became slow | Look at latency first; raising the limit may worsen the problem |
| Test is flaky | Real clocks with tiny delays | Keep delays at least a few hundred milliseconds in tests; avoid asserting exact timings |
| Duplicate payments after a timeout | Retried at another layer without idempotency | Add idempotency keys; remove retries on POST |
Cost and cleanup
This lesson costs nothing: the code and tests run locally in Docker, and no AWS resources are created. There is nothing to tear down.
# Run the tests and the demo, then clean build output.
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
-v "$PWD/labs/m12-resilient-account-api:/src" -v cea-nuget-m12:/root/.nuget/packages -w /src \
mcr.microsoft.com/dotnet/sdk:10.0 \
sh -c "dotnet test --solution ResilientAccountApi.slnx; dotnet run --project src/Harbourline.Resilience.Demo; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"Exercises
- 🟢 Guided: Run the tests and the demo. In
ResilienceOptions, changeRetryBaseDelayandMaxRetryAttemptsin the demo, run again, and explain every change in the table. - 🟡 Challenge: Add a slow-call rule: make a second test where the backend answers in 900 ms (below the attempt timeout) but your own latency target is 500 ms. Show what a breaker that counts only failures does, then change the pipeline or add a metric so that "slow" is treated as a problem. (Hint: the breaker's
ShouldHandledecides what counts.) - 🔴 Live problem: "After the Core Banking incident we saw the API make 3.8 times its normal call volume to Core Banking, and the recovery took 20 minutes." Identify the three most likely causes from this lesson (hint: layers of retries, no jitter or budget, long break/timeouts) and write the changes and the tests that would have caught them.
Assignment
Add a docs/resilience-standard.md to your portfolio repository: the six patterns, Harbourline's default numbers and why, the retry rules (which operations, which layer), the fallback register (what is shown when each dependency fails, and who approved it), and the list of tests that prove each layer. Link to your lab results.
Solutions
Raising MaxRetryAttempts increases backend calls per failed request but the retry budget (10 percent plus an initial bank of tokens) still limits the total, so in the "hard outage" row the calls fall quickly to 1. A larger RetryBaseDelay lengthens the "two blips" request. Raising the attempt timeout above the backend's 3 s delay turns the brown-out from TimeoutRejectedException into a slow success, which is exactly the trade-off between patience and protecting your threads.
A breaker that handles only failures sees 900 ms successes as healthy and never opens, so latency stays 900 ms and threads stay held. Options: lower the attempt timeout to 500 ms so slow calls become timeouts (which the breaker counts), or write a ShouldHandle predicate that treats a response whose elapsed time exceeds a threshold as a failure (Polly's circuit breaker can also be configured for slow-call detection in some designs ⚠ VERIFY), or emit a latency metric and alarm. The simplest correct answer for most services: make the timeout reflect your latency budget, so "too slow" is a failure.
Likely causes: (1) retries at several layers (client app, API, SDK) multiplying calls; (2) no jitter and no retry budget, so retries synchronised and continued while Core Banking was down; (3) long total timeout and long break duration meant threads waited and the circuit probed too eagerly or too late. Changes: retry only at the API layer (set the SDK to a small number), add jitter and a 10 percent retry budget, shorten timeouts to the latency budget, set the breaker's minimum throughput and break duration deliberately, and shed best-effort traffic. Tests: A_retry_budget_stops_retries_when_traffic_is_all_failures, the circuit open/close tests, and a bulkhead test, plus a counting assertion on ScriptedBackend.Calls in each scenario.
Knowledge check
Answer: So it limits the sum of all attempts and backoff delays. Inside the retry it would only bound a single attempt, and a call with three retries could take three times as long as intended.
Answer: The server may have processed the payment before failing, so a retry could duplicate the money movement. Unsafe methods are not retried unless the operation is made idempotent with an idempotency key and the business agrees.
Answer: Retry storms: when a dependency is failing, every request retrying multiplies the load. The budget allows retries only as a fraction of normal traffic, so once it is spent clients fail fast.
Answer: To avoid opening on statistically meaningless samples: one failure out of two calls is a 50 percent failure ratio but proves little. The breaker only evaluates the ratio once enough calls have occurred in the sampling window.
Answer: TimeoutRejectedException, not System.TimeoutException. Retry predicates and fallbacks must handle it explicitly.
Interview questions
- "Explain the order of strategies in a resilience pipeline." Bulkhead outermost to reject fast, total timeout around retries, retry around the breaker so each attempt is counted, attempt timeout innermost. Each layer protects a different resource or bounds a different quantity.
- "How do you stop retries from making an outage worse?" Backoff with jitter, a retry budget, retrying at only one layer, never retrying non-idempotent operations, and a circuit breaker so retries stop when the dependency is clearly down.
- "What is load shedding and how is it different from throttling?" Shedding refuses work you cannot afford right now, prioritising what matters, based on your own saturation. Throttling enforces a per-client limit regardless of your health.
- "When is a fallback wrong?" When it would hide a permanent error or let an unsafe decision proceed, such as approving a payment because the fraud service is down.
Certification practice
A payment API calls a downstream service that sometimes responds slowly for several minutes. During these periods the API's threads are exhausted and unrelated endpoints also fail. Which design change best isolates the failure?
A. Increase the number of threads. B. Add a bulkhead with a concurrency limit and timeouts on the downstream calls, and a circuit breaker. C. Remove all timeouts. D. Retry every failed call immediately.
Answer: B. Limiting concurrent calls and bounding waits stops the slow dependency from consuming the shared resources, and a breaker stops calling it while it is unhealthy. A delays the exhaustion. C worsens it. D multiplies load.
An application calling DynamoDB gets throttling errors during spikes. The team wants to handle this correctly without overwhelming the table. Which approach is recommended?
A. Retry immediately in a tight loop. B. Use exponential backoff with jitter, and limit the number of retries. C. Disable retries. D. Increase the client timeout to ten minutes.
Answer: B. Exponential backoff with jitter spreads retries over time and lets capacity recover. A creates a retry storm, C turns brief throttling into user-visible errors, and D hides the problem while holding resources.
An API occasionally returns duplicate records after clients retry requests that timed out. What is the most effective fix?
A. Increase the instance size. B. Make the create operation idempotent using a client-supplied idempotency key that the service de-duplicates. C. Reduce the retry count to zero for all operations. D. Add a larger database.
Answer: B. Idempotency makes a repeated request safe, which is the correct fix for ambiguous timeouts. C removes the symptom by sacrificing reliability for reads and safe writes. A and D do not address duplication.
Completion checklist
- I can name the six patterns and what each protects against
- I can explain why the layers are ordered as they are
- I ran the lab tests and the demo and can explain each row of the output
- I identified which operations in a service may be retried, and why
- I wrote down a fallback decision for each dependency of one service
Key takeaways
- Slowness is more dangerous than failure: timeouts, bulkheads and load shedding stop a slow dependency taking everything down.
- Retries need backoff, jitter, a budget, a single layer and idempotent operations.
- A circuit breaker needs a minimum throughput, a sensible break duration and an explicit fallback.
- Prove every layer with a fault-injecting test. Numbers without measurements are guesses.
Further reading
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Multi-AZ by default: RDS, Aurora, ElastiCache, ECS and your own services
Know exactly what Multi-AZ means for each service Harbourline uses, which parts AWS does for you and which parts are your job, and make your own stateless .NET services AZ-aware and failover-tolerant.
The four DR strategies: backup and restore, pilot light, warm standby and active-active
Compare the four disaster-recovery strategies against RTO, RPO and cost, choose one per business service, and understand what each demands from your infrastructure code, data replication, quotas and runbooks.