Skip to Content
Cloud Engineering Academy
12. High Availability & Disaster Recovery

Resilience patterns in .NET: timeouts, retries, breakers, bulkheads and load shedding

Make a .NET API survive a slow or failing dependency with a layered resilience pipeline, a retry budget, a circuit breaker, a bulkhead, load shedding and a graceful fallback, and prove each layer with a fault-injecting test backend.

intermediate⏱ 2 h 🟢 FreeSAA-C03DVA-C02SOA-C03

Assumes: .NET 10 LTS · Microsoft.Extensions.Http.Resilience 10.11.0 (Polly v8) · AWS SDK for .NET v4 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Classify a failure as transient, slow, permanent or overload, and pick the pattern that fits each.
  • Build a layered resilience pipeline for HttpClient with Microsoft.Extensions.Http.Resilience: bulkhead, total timeout, retry with backoff and jitter, circuit breaker and attempt timeout, in the right order.
  • Explain why retries without a budget cause outages, and add a retry budget.
  • Decide which operations may be retried (and why a payment POST must not be).
  • Add load shedding with priorities, and a fallback that returns a labelled stale value.
  • Configure AWS SDK for .NET v4 retries without multiplying them with your own.
  • Test every layer deterministically with a fault-injecting backend, with no AWS account.

Prerequisites

The code for this lesson is labs/m12-resilient-account-api (see Lab: Resilient Account API).

Why this matters in banking

At 10:02 on a Monday, Core Banking begins answering balance requests in 6 seconds instead of 80 milliseconds. Nothing has crashed; every health check passes. Here is what happens in a service with no resilience patterns:

  1. Each request now holds a thread and a connection for 6 seconds.
  2. The thread pool and the HTTP connection pool fill up. New requests queue.
  3. Customer apps time out after 10 seconds and retry, doubling the load.
  4. The API's own health check, which is unrelated to Core Banking, now times out because there are no free threads. The load balancer removes tasks. The remaining tasks take all the traffic and fail faster.
  5. A slow dependency has become a complete outage of the API, including endpoints that never use Core Banking.

This is a cascading failure, and slowness is more dangerous than failure, because failures are fast. The patterns in this lesson exist to turn "slow dependency" into "some features degrade, the rest keep working".

The idea in plain language

A resilience pattern is a rule for how your code behaves when a call does not go well. You already know the cousins from .NET: HttpClient.Timeout, CancellationToken, Polly, WCF's OperationTimeout, and ReliableSession. The AWS-specific part is small: the SDK has its own retry settings, and AWS services throttle and fail over in ways you must tolerate.

Six patterns do nearly all the work:

PatternOne-sentence ideaProtects against
TimeoutStop waiting after a limitSlow dependencies holding your resources
Retry (backoff and jitter)Try again, with growing and randomised delaysBrief, transient faults
Circuit breakerAfter repeated failures, stop calling for a whileHammering a dependency that is down
BulkheadLimit how much of your capacity one dependency may useOne slow dependency exhausting everything
Load sheddingRefuse some work early to keep serving the restOverload
FallbackReturn something useful (stale, default) instead of failingFailures you can degrade gracefully

Analogy: a bank branch handles a broken card reader like this. The cashier gives up on a card after 10 seconds (timeout), tries once more after a pause (retry), and after three failures puts a "reader out of order" sign up and stops trying for 15 minutes (breaker). Only two counters are allowed to deal with card problems so the others keep serving cash (bulkhead). When the queue goes out the door, the manager lets the "pay a bill" customers in and asks the "open a savings account" customers to come back later (load shedding). A customer who only wants their balance is shown the last receipt, clearly dated (fallback).

Classify the failure first

FailureLooks likeWhat helpsWhat hurts
TransientA single 503, a dropped connection, a throttle (HTTP 429 or an AWS throttling exception)Retry with backoff and jitterImmediate retries; unbounded retries
Slow (brown-out)Responses take seconds, no errorsTimeouts, bulkhead, breaker (slow-call), load sheddingLong timeouts; retrying slow calls
Permanent404, 400, validation error, a bugDo not retry. Return the answerRetrying; masking with a fallback
OverloadMany callers, saturated serviceBackoff, retry budget, load shedding, rate limitingRetry storms
OutageEverything fails fastCircuit breaker, fallback, failoverRetrying every call

The three rules that keep retries safe

  1. Only retry what is safe to repeat. A GET is naturally idempotent. A POST that moves money is not. Either do not retry it, or make it idempotent with an idempotency key that the server de-duplicates. The lab pipeline never retries POST, PUT, PATCH or DELETE.
  2. Add jitter. If 1,000 clients all retry after exactly 200 ms, then 400 ms, they hit the service in synchronised waves. Randomising each delay spreads the load. Exponential backoff plus jitter is the standard.
  3. Cap retries as a share of traffic. If every request retries twice and the dependency is down, you send three times the load to something already failing. A retry budget says "retries may be at most 10 percent of requests". When the budget is spent, fail fast.

A related hazard is retry amplification across layers. If the browser retries 3 times, the API retries 3 times, and the AWS SDK retries 3 times, one click becomes 27 calls. Retry at one layer, usually the one closest to the user-visible operation, and tune the others down.

What problem the AWS service solves

This lesson is mostly .NET, but the AWS context shapes the design:

  • AWS services throttle and fail over. DynamoDB, SQS and others return throttling errors under load, and databases fail over in tens of seconds (Multi-AZ by default). Your code must expect brief errors.
  • The AWS SDK for .NET has built-in retries. Per the SDK v4 developer guide, the retry mode is controlled by RetryMode (Legacy, Standard, Adaptive which is documented as experimental) and MaxErrorRetry, which can also come from AWS_RETRY_MODE and AWS_MAX_ATTEMPTS or the shared config file. If nothing is set, the SDK defaults to Legacy mode with a default MaxErrorRetry that the documentation lists as 4 for Legacy and 2 for Standard ⚠ VERIFY the numbers for your SDK version. Retrying again in Polly on top of this multiplies attempts.
  • SDK timeouts: for async calls, use the CancellationToken parameter to impose a deadline. The Timeout property on the client config does not affect async calls, per the same guide.
  • The Polly-based libraries are the Microsoft-supported way to add the other patterns to HttpClient.

Limitations to be honest about: patterns reduce risk but do not remove it. A breaker can open on a false alarm. A fallback can serve stale data for too long. Bulkhead sizing is a guess until you measure. All of these need monitoring and tuning, and every parameter in the lab is a starting point, not a recommendation.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS / .NET equivalentWhat's different
Polly (v7 policies)Polly v8 ResiliencePipelinePipelines replace policies; strategies are added in order, outermost first
Microsoft.Extensions.Http.PollyMicrosoft.Extensions.Http.ResilienceThe new package; AddStandardResilienceHandler is the one-liner
Azure SDK built-in retry (RetryOptions)AWS SDK RetryMode and MaxErrorRetryBoth retry internally; avoid stacking your own on top blindly
Azure API Management retry, rate-limit policiesAPI Gateway throttling, WAF rate rules, your own RateLimiterEdge limits protect you; in-process limiters protect dependencies
Azure Front Door/App Gateway circuit breaking (limited)In-process breakers; Envoy/App Mesh outlier detection if using a meshBreakers usually live in the client library
System.Threading.RateLimitingSame library, used by Polly's concurrency limiterThe bulkhead is a concurrency limiter
Polly Fallback policyPolly FallbackStrategyOptions or plain try/catchA fallback is a business decision as much as a code pattern

Architecture

The lab pipeline wraps every call to Core Banking. The first layer added is the outermost.

The circuit breaker has three states:

In the Open state calls fail immediately without touching the dependency, which both protects it (it can recover) and protects you (no threads wasted waiting).

Step-by-step implementation

Add the packages

dotnet add package Microsoft.Extensions.Http.Resilience --version 10.11.0

The package brings in Polly v8 and its rate-limiting strategy. For tests, the lab uses a scripted HttpMessageHandler, so no extra mocking library is required.

Start from the standard handler, then own the pipeline

For a first project, AddStandardResilienceHandler() gives you a sensible pipeline in one line. The documented defaults are: rate limiter permit 1,000 with queue 0; total timeout 30 s; retry max 3, exponential, jitter on, base delay 2 s; circuit breaker failure ratio 10%, minimum throughput 100, sampling 30 s, break 5 s; attempt timeout 10 s. Those values suit a generic internet call, not a balance lookup that must answer in 200 ms. For Harbourline we replace them with our own pipeline (AddResilienceHandler) so that every number is a decision.

Order the layers deliberately

Outermost to innermost: bulkhead, total timeout, retry, breaker, attempt timeout. The total timeout must be outside the retry (so it caps all attempts); the attempt timeout must be inside it (so each try is bounded); the breaker sits inside the retry so each attempt is counted, and an open circuit short-circuits quickly. Put the bulkhead outermost so rejected calls cost almost nothing.

Make retries safe

Retry only transient results (connection errors, our own timeouts, 408, 429, 5xx). Never retry unsafe HTTP methods. Add a retry budget. Do not retry an open circuit.

Add load shedding in front

Classify requests (payments and health are critical; reports are best-effort) and refuse low-priority work first when in-flight requests approach capacity. Return 503 with Retry-After.

Decide your fallbacks

For each dependency, write one line: "if this fails we show ___ or we return ___". For balances, the answer is the last known balance labelled with its time. For "is there enough money to pay", the answer is fail (no fallback).

Prove it

Write a test per layer against a fault-injecting backend, then run the demo and read the table. If you cannot make each layer visibly fire, you do not know that it works.

AWS Console instructions

The patterns live in code, but you observe them in the console:

  • CloudWatch metrics for your own breaker and retry counters: Console → CloudWatch → Metrics → All metrics → your custom namespace (Module 11 shows how to emit them).
  • Throttling you are retrying through: Console → CloudWatch → Metrics → DynamoDB (or SQS) → Table/Queue metrics and look for throttled-request metrics ⚠ VERIFY the exact metric names for your service.
  • Service quotas: Console → Service Quotas to see the limits behind throttling errors and request increases before peaks.

AWS CLI

PROFILE=learn-sandbox

# Show the SDK/CLI retry settings in effect for the profile. Unset values mean the defaults.
aws configure get retry_mode --profile $PROFILE
aws configure get max_attempts --profile $PROFILE

# Set retry behaviour for tools that read the shared config (the AWS SDK for .NET reads these too).
aws configure set retry_mode standard --profile $PROFILE
aws configure set max_attempts 3 --profile $PROFILE

# The same through environment variables (apply to the .NET SDK as well):
export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=3

max_attempts counts total attempts in the CLI, whereas MaxErrorRetry in the .NET SDK counts retries after the first attempt; keep this difference in mind when comparing numbers ⚠ VERIFY for your versions.

Infrastructure as Code

Resilience patterns are application code, so there is no CDK or Terraform for the patterns themselves. The infrastructure that supports them is in the lab stack: the ReadinessGate and health-check settings, the AZ spread and the scaling policies (Auto Scaling everywhere). One thing you can configure in infrastructure is the SDK retry behaviour for a container, through environment variables:

// On the ECS container definition: make the AWS SDK retry once and let your own pipeline own the rest.
taskDefinition.AddContainer("api", new ContainerDefinitionOptions
{
    Image = ContainerImage.FromRegistry(props.ContainerImage),
    Environment = new Dictionary<string, string>
    {
        ["AWS_RETRY_MODE"] = "standard",
        ["AWS_MAX_ATTEMPTS"] = "3", // check how your SDK version counts attempts ⚠ VERIFY
    },
});

Complete code

The files below are in labs/m12-resilient-account-api/src/Harbourline.Resilience. Together with the tests they run on any machine with Docker.

ResilienceOptions.cs
namespace Harbourline.Resilience;

public sealed class ResilienceOptions
{
    public int MaxConcurrentCalls { get; set; } = 50;          // bulkhead
    public int QueuedCalls { get; set; }                       // 0 = reject immediately
    public TimeSpan TotalTimeout { get; set; } = TimeSpan.FromSeconds(8);
    public TimeSpan AttemptTimeout { get; set; } = TimeSpan.FromSeconds(2);
    public int MaxRetryAttempts { get; set; } = 2;
    public TimeSpan RetryBaseDelay { get; set; } = TimeSpan.FromMilliseconds(200);
    public RetryBudget? RetryBudget { get; set; }
    public double BreakerFailureRatio { get; set; } = 0.5;
    public int BreakerMinimumThroughput { get; set; } = 20;
    public TimeSpan BreakerSamplingDuration { get; set; } = TimeSpan.FromSeconds(30);
    public TimeSpan BreakerBreakDuration { get; set; } = TimeSpan.FromSeconds(15);

    public void Validate()
    {
        if (MaxConcurrentCalls < 1) throw new ArgumentException("MaxConcurrentCalls must be at least 1.", nameof(MaxConcurrentCalls));
        if (MaxRetryAttempts < 0) throw new ArgumentException("MaxRetryAttempts cannot be negative.", nameof(MaxRetryAttempts));
        if (AttemptTimeout <= TimeSpan.Zero) throw new ArgumentException("AttemptTimeout must be positive.", nameof(AttemptTimeout));
        if (TotalTimeout <= AttemptTimeout) throw new ArgumentException("TotalTimeout must be longer than AttemptTimeout, otherwise retries can never happen.", nameof(TotalTimeout));
        if (BreakerFailureRatio is <= 0.0 or > 1.0) throw new ArgumentException("BreakerFailureRatio must be in (0, 1].", nameof(BreakerFailureRatio));
        if (BreakerMinimumThroughput < 2) throw new ArgumentException("BreakerMinimumThroughput must be at least 2.", nameof(BreakerMinimumThroughput));
    }
}
HarbourlineResilience.cs
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Polly;
using Polly.Retry;
using Polly.Timeout;

namespace Harbourline.Resilience;

public static class HarbourlineResilience
{
    public const string PipelineName = "harbourline-core-banking";

    public static IHttpClientBuilder AddHarbourlineResilience(this IHttpClientBuilder builder, ResilienceOptions options)
    {
        options.Validate();

        builder.AddResilienceHandler(PipelineName, pipeline =>
        {
            // 1. Bulkhead: cap concurrent calls.
            pipeline.AddConcurrencyLimiter(options.MaxConcurrentCalls, options.QueuedCalls);

            // 2. Total timeout: one logical call never takes longer than this, retries included.
            pipeline.AddTimeout(options.TotalTimeout);

            // 3. Retry with exponential backoff + jitter, only for transient faults and safe methods.
            if (options.MaxRetryAttempts > 0)
            {
                var retry = new HttpRetryStrategyOptions
                {
                    MaxRetryAttempts = options.MaxRetryAttempts,
                    BackoffType = DelayBackoffType.Exponential,
                    UseJitter = true,
                    Delay = options.RetryBaseDelay,
                    ShouldHandle = args => ValueTask.FromResult(ShouldRetry(args, options.RetryBudget)),
                };
                retry.DisableForUnsafeHttpMethods(); // must come AFTER ShouldHandle is set
                pipeline.AddRetry(retry);
            }

            // 4. Circuit breaker.
            pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
            {
                FailureRatio = options.BreakerFailureRatio,
                MinimumThroughput = options.BreakerMinimumThroughput,
                SamplingDuration = options.BreakerSamplingDuration,
                BreakDuration = options.BreakerBreakDuration,
            });

            // 5. Attempt timeout: each try has its own, shorter, limit.
            pipeline.AddTimeout(new HttpTimeoutStrategyOptions { Timeout = options.AttemptTimeout });
        });

        return builder;
    }

    private static bool ShouldRetry(RetryPredicateArguments<HttpResponseMessage> args, RetryBudget? budget)
    {
        if (args.AttemptNumber == 0)
        {
            budget?.RecordRequest(); // each logical request earns a little retry allowance
        }

        return IsTransient(args.Outcome) && (budget?.TryWithdraw() ?? true);
    }

    internal static bool IsTransient(Outcome<HttpResponseMessage> outcome)
    {
        if (outcome.Exception is not null)
        {
            return outcome.Exception is HttpRequestException or TimeoutRejectedException;
        }

        return outcome.Result is { } response && (int)response.StatusCode is 408 or 429 or >= 500;
    }
}
RetryBudget.cs
namespace Harbourline.Resilience;

public sealed class RetryBudget
{
    private readonly object _gate = new();
    private readonly double _retryRatio;
    private readonly double _maxTokens;
    private double _tokens;

    public RetryBudget(double retryRatio = 0.1, double maxTokens = 10)
    {
        if (retryRatio is <= 0.0 or > 1.0) throw new ArgumentOutOfRangeException(nameof(retryRatio));
        if (maxTokens < 1.0) throw new ArgumentOutOfRangeException(nameof(maxTokens));
        _retryRatio = retryRatio;
        _maxTokens = maxTokens;
        _tokens = maxTokens;
    }

    public void RecordRequest() { lock (_gate) { _tokens = Math.Min(_maxTokens, _tokens + _retryRatio); } }

    public bool TryWithdraw()
    {
        lock (_gate)
        {
            if (_tokens < 1.0) return false;
            _tokens -= 1.0;
            return true;
        }
    }
}
LoadShedder.cs (core)
public sealed class LoadShedder
{
    private readonly int _capacity, _normalLimit, _bestEffortLimit;
    private int _inFlight;

    public LoadShedder(int capacity, double normalShare = 0.8, double bestEffortShare = 0.5)
    {
        _capacity = capacity;
        _normalLimit = Math.Max(1, (int)Math.Ceiling(capacity * normalShare));
        _bestEffortLimit = Math.Max(1, (int)Math.Ceiling(capacity * bestEffortShare));
    }

    public Lease? TryEnter(RequestPriority priority)
    {
        var limit = priority switch
        {
            RequestPriority.Critical => _capacity,
            RequestPriority.Normal => _normalLimit,
            _ => _bestEffortLimit,
        };

        if (Interlocked.Increment(ref _inFlight) > limit)
        {
            Interlocked.Decrement(ref _inFlight);
            return null; // shed
        }

        return new Lease(this);
    }
    // Lease.Dispose() decrements _inFlight exactly once (full file in the lab).
}

The balance read with graceful degradation:

BalanceService.cs (core)
public async Task<BalanceView?> GetAsync(string accountId, CancellationToken cancellationToken = default)
{
    try
    {
        var result = await client.GetBalanceAsync(accountId, cancellationToken);
        var view = new BalanceView(result.AccountId, result.Balance, result.Currency, clock.GetUtcNow(), IsStale: false);
        _lastKnown[accountId] = view;
        return view;
    }
    catch (Exception ex) when (IsDependencyFailure(ex))
    {
        return _lastKnown.TryGetValue(accountId, out var last) ? last with { IsStale = true } : null;
    }
}

// A timeout, an open circuit, a full bulkhead, a connection error or an HTTP 5xx may be covered by a stale answer.
// A 404 is an ANSWER ("no such account") and is never hidden.
private static bool IsDependencyFailure(Exception ex) => ex switch
{
    TimeoutRejectedException or BrokenCircuitException or RateLimiterRejectedException => true,
    HttpRequestException { StatusCode: null } => true,
    HttpRequestException { StatusCode: { } status } => (int)status >= 500,
    _ => false,
};

And the fault-injecting backend that makes all of it testable:

ScriptedBackend.cs (core)
public sealed record Fault(HttpStatusCode Status = HttpStatusCode.OK, TimeSpan Delay = default, bool ConnectionReset = false)
{
    public static Fault Ok() => new();
    public static Fault Error(HttpStatusCode status) => new(status);
    public static Fault Slow(TimeSpan delay) => new(Delay: delay);
    public static Fault Reset() => new(ConnectionReset: true);
}

// Call 1 gets the first fault, call 2 the second, and so on; the LAST fault repeats forever.
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
{
    var callNumber = Interlocked.Increment(ref _calls);
    var fault = _script[Math.Min(callNumber, _script.Length) - 1];

    if (fault.Delay > TimeSpan.Zero) await Task.Delay(fault.Delay, ct);
    if (fault.ConnectionReset) throw new HttpRequestException("Connection reset by peer (injected fault).");

    return new HttpResponseMessage(fault.Status)
    {
        Content = new StringContent(BalanceJson, Encoding.UTF8, "application/json"),
        RequestMessage = request,
    };
}

A test that proves the circuit breaker fires and recovers:

ResiliencePipelineTests.cs (excerpt)
[Fact]
public async Task The_circuit_opens_after_repeated_failures_and_stops_calling_the_backend()
{
    var backend = new ScriptedBackend(Fault.Error(HttpStatusCode.InternalServerError));
    var options = FastOptions();           // MinimumThroughput = 4, BreakDuration = 600 ms
    options.MaxRetryAttempts = 0;          // one backend call per logical call keeps the arithmetic obvious
    await using var provider = Build(backend, options);
    var client = provider.GetRequiredService<CoreBankingClient>();

    for (var i = 0; i < 4; i++)
    {
        await Assert.ThrowsAsync<HttpRequestException>(() => client.GetBalanceAsync("ACC-0000000001"));
    }
    Assert.Equal(4, backend.Calls);

    // The circuit is now open: calls fail immediately and never reach the backend.
    await Assert.ThrowsAsync<BrokenCircuitException>(() => client.GetBalanceAsync("ACC-0000000001"));
    Assert.Equal(4, backend.Calls);
}

What the demo shows

dotnet run --project src/Harbourline.Resilience.Demo replays four scenarios. This is real output from one run (timings vary):

Scenario                               Call  Outcome                            Backend calls     Time
--------------------------------------------------------------------------------------------------------
Healthy                                   1  OK 125.40 GBP                                  1    218 ms
                                          2  OK 125.40 GBP                                  1      1 ms
                                          3  OK 125.40 GBP                                  1      0 ms

Two blips, then healthy                   1  OK 125.40 GBP                                  3    169 ms
                                          2  OK 125.40 GBP                                  1      0 ms
                                          3  OK 125.40 GBP                                  1      0 ms

Brown-out (every answer takes 3 s)        1  TimeoutRejectedException                       3   3231 ms
                                          2  TimeoutRejectedException                       2   2117 ms

Hard outage (always HTTP 500)             1  HttpRequestException                           3    213 ms
                                          2  HttpRequestException                           2     41 ms
                                          3  HttpRequestException                           1      4 ms
                                          4  BrokenCircuitException                         0      0 ms
                                          5  BrokenCircuitException                         0      0 ms
                                          ...

Code explained line by line

  1. AddResilienceHandler(PipelineName, pipeline => ...) registers a named pipeline on the HttpClient. Strategies are applied in the order added, outermost first.
  2. AddConcurrencyLimiter(max, queue) is the bulkhead. With a queue of 0, call number max + 1 is rejected immediately with RateLimiterRejectedException. Rejecting fast is better than queuing behind a slow dependency.
  3. AddTimeout(options.TotalTimeout) placed before the retry caps the whole call. Placed after the retry it would only cap one attempt. The demo's brown-out row shows it: three attempts of 1 s each, about 3.2 s in total, below the 4 s total timeout.
  4. HttpRetryStrategyOptions with BackoffType = Exponential and UseJitter = true produces delays that grow and are randomised.
  5. ShouldHandle runs after each attempt. Our predicate records the request on AttemptNumber == 0, then checks the outcome is transient, then asks the budget for permission. In the "hard outage" row the backend calls fall from 3 to 2 to 1 as the budget empties: the budget is visibly doing its job.
  6. DisableForUnsafeHttpMethods() wraps the predicate so POST, PUT, PATCH, DELETE and CONNECT are never retried. It must be called after ShouldHandle is assigned, or it wraps the default and is then overwritten.
  7. HttpCircuitBreakerStrategyOptions opens when the failure ratio in the sampling window passes the threshold and at least MinimumThroughput calls were seen. In the demo, after six backend calls the circuit opens and calls 4 and later cost zero backend calls and zero milliseconds.
  8. HttpTimeoutStrategyOptions is the attempt timeout, placed innermost so each try is bounded. A cancelled attempt surfaces as TimeoutRejectedException, which is not the standard TimeoutException. The retry predicate and the fallback must handle it explicitly.
  9. IsTransient treats connection errors, our timeouts and HTTP 408, 429 and 5xx as retryable, and deliberately does not retry BrokenCircuitException: retrying an open circuit just adds latency.
  10. LoadShedder.TryEnter increments the in-flight counter first and backs out if over the limit: a lock-free "try acquire". Critical traffic may use all slots; normal 80 percent; best-effort 50 percent.
  11. BalanceService.IsDependencyFailure limits the fallback to genuine dependency failures. A 404 must reach the caller as "no such account".
  12. ScriptedBackend is a deterministic fault injector. Because the last fault repeats, new ScriptedBackend(Fault.Error(500)) is "always failing" and new ScriptedBackend(Fault.Error(503), Fault.Reset(), Fault.Ok()) is "two blips, then fine".

Security considerations

  • Fail closed for security decisions. A fallback may show a stale balance. It must never turn an authorisation failure into "allowed". If the sanctions-screening or fraud service is down, a payment waits or is rejected; it is not approved by default.
  • Idempotency is a security and integrity control. Retried or replayed requests must not create two payments. Require Idempotency-Key on money-moving POSTs, store results, and return the stored result for a repeat.
  • Do not log sensitive request bodies in retry telemetry. Log the correlation ID, attempt number and status code, not account numbers or payloads.
  • Breakers and shedding are availability controls for you, and attack surface too. An attacker may try to trip a breaker or fill the bulkhead to degrade service for others. Rate limit per client at the edge (WAF, API Gateway) so one caller cannot exhaust shared capacity.
  • Stale data must be labelled. Showing a balance without its time can mislead a customer into overspending. BalanceView.AsOf exists for this reason, and compliance should agree the wording.

⚠ Learning shortcut — not for production

The lab keeps the last known balances in an in-memory dictionary on one process, and its breaker state is per process. Production uses a shared cache (ElastiCache or DynamoDB) with a maximum age, per-dependency pipelines sized from measurements, telemetry exported to CloudWatch or OpenTelemetry, and a documented decision (with the business and compliance) on when stale data may be shown. Also, the demo's timings and thresholds are chosen to finish in seconds, not to be correct for your service.

Banking use case

Harbourline's Accounts API serves a balance in under 200 ms at the 99th percentile on a normal day. The resilience design agreed with the service owner:

DependencyTimeout (attempt/total)RetriesBreakerBulkheadFallback
Core Banking balance (GET)400 ms / 1.5 s1, jittered, budget 10%50% over 30 s, min 20 calls, 15 s break50 in flightLast known balance, labelled "as of"
Core Banking transfer (POST)3 s / 3 sNone; idempotency key and a status lookupShared with balance? No, separate pipeline20 in flightNone: return 202 with a "pending" status or an error
Sanctions screening2 s / 5 s2 with backoff50% over 30 s10 in flightNone: hold the payment for manual review
Notification service1 s / 3 s2Open fast20 in flightQueue for later (SQS)

Load shedding: /payments critical, /accounts normal, /reports best-effort. All numbers are fictional examples chosen to show the shape of a decision, not values to copy.

Common mistakes

  • Retrying everything. Retrying a 400, a 404 or a payment POST. Retry only transient failures of safe or idempotent operations.
  • Retries on every layer. Browser, API, SDK and gateway each retry three times and one request becomes dozens. Pick one layer.
  • No jitter. Synchronised retry waves can keep a recovering service down.
  • Timeout longer than the caller's timeout. The caller has already given up and retried while you are still working.
  • Breaker with no minimum throughput. One failure out of two opens it. Always set a sensible minimum.
  • A bulkhead that queues. A long queue behind a slow dependency is a slow outage. Reject fast.
  • Catching TimeoutException and missing TimeoutRejectedException. Polly throws its own type.
  • Fallbacks that hide permanent errors or approve risky actions. Fall back only for reads where staleness is acceptable.
  • Tuning from guesses. Without latency histograms and error rates, every number is folklore.

Troubleshooting

SymptomLikely causeFix
Retries never happenTotalTimeout shorter than AttemptTimeout, or retry only wired for a different handler nameValidate() rejects the first; check the pipeline is added to the right HttpClient
POSTs are retriedDisableForUnsafeHttpMethods() called before ShouldHandle was set, or a second handler was addedSet ShouldHandle first; use one resilience handler per client
Circuit never opensMinimumThroughput too high for the traffic, or failures not matching the breaker's handled outcomesLower the minimum; check the sampling duration; verify 5xx and timeouts are handled
Circuit opens on a single failureMinimumThroughput very lowRaise it (at least 10 to 20 for real traffic)
BrokenCircuitException foreverBreak duration too long or the probe keeps failingCheck the dependency; shorten the break; look at half-open probe results
RateLimiterRejectedException spikesBulkhead too small, or the dependency became slowLook at latency first; raising the limit may worsen the problem
Test is flakyReal clocks with tiny delaysKeep delays at least a few hundred milliseconds in tests; avoid asserting exact timings
Duplicate payments after a timeoutRetried at another layer without idempotencyAdd idempotency keys; remove retries on POST

Cost and cleanup

This lesson costs nothing: the code and tests run locally in Docker, and no AWS resources are created. There is nothing to tear down.

# Run the tests and the demo, then clean build output.
MSYS_NO_PATHCONV=1 docker run --rm --memory 3g \
  -v "$PWD/labs/m12-resilient-account-api:/src" -v cea-nuget-m12:/root/.nuget/packages -w /src \
  mcr.microsoft.com/dotnet/sdk:10.0 \
  sh -c "dotnet test --solution ResilientAccountApi.slnx; dotnet run --project src/Harbourline.Resilience.Demo; find . -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} +"

Exercises

  • 🟢 Guided: Run the tests and the demo. In ResilienceOptions, change RetryBaseDelay and MaxRetryAttempts in the demo, run again, and explain every change in the table.
  • 🟡 Challenge: Add a slow-call rule: make a second test where the backend answers in 900 ms (below the attempt timeout) but your own latency target is 500 ms. Show what a breaker that counts only failures does, then change the pipeline or add a metric so that "slow" is treated as a problem. (Hint: the breaker's ShouldHandle decides what counts.)
  • 🔴 Live problem: "After the Core Banking incident we saw the API make 3.8 times its normal call volume to Core Banking, and the recovery took 20 minutes." Identify the three most likely causes from this lesson (hint: layers of retries, no jitter or budget, long break/timeouts) and write the changes and the tests that would have caught them.

Assignment

Add a docs/resilience-standard.md to your portfolio repository: the six patterns, Harbourline's default numbers and why, the retry rules (which operations, which layer), the fallback register (what is shown when each dependency fails, and who approved it), and the list of tests that prove each layer. Link to your lab results.

Solutions

Knowledge check

Interview questions

  • "Explain the order of strategies in a resilience pipeline." Bulkhead outermost to reject fast, total timeout around retries, retry around the breaker so each attempt is counted, attempt timeout innermost. Each layer protects a different resource or bounds a different quantity.
  • "How do you stop retries from making an outage worse?" Backoff with jitter, a retry budget, retrying at only one layer, never retrying non-idempotent operations, and a circuit breaker so retries stop when the dependency is clearly down.
  • "What is load shedding and how is it different from throttling?" Shedding refuses work you cannot afford right now, prioritising what matters, based on your own saturation. Throttling enforces a per-client limit regardless of your health.
  • "When is a fallback wrong?" When it would hide a permanent error or let an unsafe decision proceed, such as approving a payment because the fraud service is down.

Certification practice

Completion checklist

  • I can name the six patterns and what each protects against
  • I can explain why the layers are ordered as they are
  • I ran the lab tests and the demo and can explain each row of the output
  • I identified which operations in a service may be retried, and why
  • I wrote down a fallback decision for each dependency of one service

Key takeaways

  • Slowness is more dangerous than failure: timeouts, bulkheads and load shedding stop a slow dependency taking everything down.
  • Retries need backoff, jitter, a budget, a single layer and idempotent operations.
  • A circuit breaker needs a minimum throughput, a sensible break duration and an explicit fallback.
  • Prove every layer with a fault-injecting test. Numbers without measurements are guesses.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page