Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Retry strategies: backoff, jitter and avoiding retry storms

Design retries that help rather than hurt: classify errors, use exponential backoff with full jitter, configure AWS SDK for .NET retry modes, line up retries across SQS, Lambda, SNS, EventBridge and Step Functions, and stop layered retries from turning an outage into a retry storm.

intermediate⏱ 2 h 🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS 4.1.0.2) · Polly.Core 8.9.1 · Microsoft.Extensions.Http.Resilience 10.11.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Sort a failure into transient, throttling or permanent, and decide whether it should be retried at all.
  • Explain exponential backoff with full jitter and calculate the delay for a given attempt.
  • Configure the AWS SDK for .NET v4 retry mode and attempt count, in code and through AWS_RETRY_MODE / AWS_MAX_ATTEMPTS, and explain the retry quota (token bucket).
  • Describe the built-in retry behaviour of SQS, Lambda (async and event source mappings), SNS, EventBridge and Step Functions, and where each one hands up when it gives up.
  • Spot retry amplification (layered retries multiplying load) and fix it with a single retry owner, retry budgets, timeouts and circuit breakers.
  • Build an SQS consumer in C# that backs off per message with ChangeMessageVisibility instead of hammering a failing dependency.

Prerequisites

  • The SQS, dead-letter-queue, SNS, EventBridge and Step Functions lessons earlier in this module (see the module overview)
  • Module 3: .NET on AWS — Worker Services, dependency injection and Lambda basics
  • Tools: .NET 10 SDK, AWS CLI v2 with the learn-sandbox SSO profile

Why this matters in banking

On a Friday afternoon, Harbourline's sanctions-screening provider slows down. Response times go from 80 ms to 9 seconds, then the provider starts returning HTTP 503.

Every layer of the payments platform "helpfully" retries:

  • The mobile app retries the payment API three times.
  • The Payments API retries the screening call three times, through Polly.
  • The AWS SDK inside the screening adapter retries the SQS send three times.
  • The Lambda function behind the queue fails, so SQS redelivers the message five times before the dead-letter queue.

One customer tap has become up to 3 × 3 × 5 = 45 screening calls. Across 4,000 payments a minute, the provider now receives roughly 180,000 calls a minute. That is the moment it was least able to cope. The provider's own rate limiter starts rejecting everyone, including Harbourline's other banks' traffic. The incident lasts two hours instead of ten minutes.

Meanwhile some payments were retried after a timeout that had actually succeeded, so a few customers are debited twice.

Retries are the most common resilience tool and the most common cause of self-inflicted outages. A bank must know:

  • which layer owns retries;
  • how long it waits between attempts;
  • when it gives up;
  • that every retried operation is idempotent.

The idea in plain language

A retry means "try the same request again because the failure was probably temporary". Three ideas make retries safe:

  1. Only retry what might succeed next time. A network blip, an HTTP 503 or a throttling error might go away. A validation error, an AccessDenied or "insufficient funds" will not.
  2. Wait longer each time, and wait a random amount. Exponential backoff doubles the wait after each failure (100 ms, 200 ms, 400 ms…). Jitter picks a random point within that wait, so thousands of clients that failed at the same moment do not all retry at the same moment.
  3. Give up, and hand over. After a small number of attempts, or once a time budget is spent, stop. Put the work somewhere safe, such as a dead-letter queue or a failed-payment state, rather than retrying forever.

Analogy: You ring a branch and the line is busy. You do not redial every second, because that ties up the switchboard for everyone. You wait a minute, then a few minutes, then you send an email instead. If every customer redialled on the same second after the same "please call back in one minute" message, the switchboard would stay jammed. Jitter is everyone choosing a slightly different time to call back.

Two more terms you will see:

  • Retry storm (retry amplification): retries at several layers multiply. The total load on a failing dependency grows exactly when it can least handle it.
  • Retry budget (retry quota): a cap on how many retries a client may make relative to its successful requests. When the budget is spent, the client fails fast instead of piling on.

What problem AWS's retry features solve

AWS builds retries into almost every layer, and you rarely need to write a retry loop around an AWS API call yourself. What you do need is to know each layer's behaviour so they don't stack.

LayerWho retriesDefault behaviour (checked 2026-11-18)When it gives up
AWS SDK for .NET (any AWS API call)The SDK clientTransient and throttling errors retried with backoff and jitter; MaxErrorRetry 2 in standard mode (3 attempts in total). See the SDK section below for the default-mode caveatThrows the last exception to your code
SQS consumer (your poller or Lambda)SQS redelivers when the visibility timeout expiresRedelivered until maxReceiveCount on the redrive policy, or until the retention period endsMoves to the DLQ (if configured), otherwise deleted at end of retention
Lambda async invoke (S3, SNS, EventBridge → Lambda)Lambda's internal queue2 retries for function errors (about 1 minute, then 2 minutes apart); throttles and system errors retried for up to 6 hoursOn-failure destination or DLQ, otherwise discarded
Lambda event source mapping for SQSSQS (through visibility timeout)Whole batch returns unless you report partial batch failuresQueue's redrive policy
Lambda event source mapping for Kinesis / DynamoDB StreamsLambdaRetries until the record expires by default (MaximumRetryAttempts -1) — blocks the shardOn-failure destination after the limits you set
SNS → SQS / LambdaSNSAWS-managed endpoints: 100,015 attempts over 23 daysSubscription DLQ, otherwise discarded
SNS → HTTP/SSNSCustomisable delivery policy; total retry time at most 3,600 secondsSubscription DLQ, otherwise discarded
EventBridge rule target (Classic bus)EventBridge24 hours and up to 185 attempts, exponential backoff with jitterTarget DLQ (standard SQS queue), otherwise dropped
Step Functions TaskYour Retry blockNo retry unless you declare one; defaults when you do: IntervalSeconds 1, MaxAttempts 3, BackoffRate 2.0, JitterStrategy NONECatch block, otherwise the execution fails

Limitations to know:

  • Retries cost money and time. Every Lambda retry is a billed invocation. Every Step Functions retry is a billed state transition in Standard workflows. Every SQS redelivery is another ReceiveMessage.
  • Not every retry is safe. A retried SendMessage to a standard queue can create a duplicate message. A retried payment call can create a duplicate payment. Retries assume idempotency, which is covered in Idempotency in .NET.
  • Retrying hides problems. A service that only works on the third attempt is broken. Measure retry counts and alarm on them.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Azure SDK RetryOptions (Mode = Exponential, MaxRetries)AWS SDK for .NET RetryMode and MaxErrorRetry on the client configAWS also has a cross-SDK retry quota that stops retrying when most calls fail
Polly / Microsoft.Extensions.Http.ResilienceSame libraries — you still use them for your own HTTP callsDo not wrap AWS SDK calls in Polly retries as well; the SDK already retries
Service Bus MaxDeliveryCountSQS redrive policy maxReceiveCountSQS retry spacing is controlled by the visibility timeout, not a lock duration
Service Bus scheduled redelivery / abandon with delayChangeMessageVisibility to push a message's next attempt into the futurePer-message, up to 12 hours from the original receive
Durable Functions RetryOptionsStep Functions Retry with BackoffRate, MaxDelaySeconds, JitterStrategyDeclared in the state machine definition, visible in the console's execution history
Event Grid retry policy (24 hours, 30 attempts)EventBridge target retry policy (24 hours, 185 attempts by default)Both configurable per subscription/target; both need a dead-letter destination to keep failures
Azure Functions host retry policiesLambda async retry settings and event source mapping settingsDifferent knobs per event source type

Architecture

The rule that prevents retry storms is one retry owner per hop. The diagram shows the Harbourline payment-screening path with retries placed deliberately.

Read it like this:

  • The app does not auto-retry money movements. If the user resubmits, the same Idempotency-Key makes it safe.
  • The API → SQS hop is an AWS API call, so the SDK owns retries there.
  • The worker → provider hop is owned by SQS redelivery. The worker makes one attempt per receive, with a short timeout. On failure it leaves the message in the queue and sets the next visibility timeout using backoff with jitter.
  • A circuit breaker in the worker stops calling the provider at all while it is clearly down. Messages simply wait in the queue, which is the cheapest buffer you have.
  • After six receives the message goes to the DLQ, and a person looks at it.

Choosing what to retry

Classify before you retry. This table is a starting point for Harbourline services.

ErrorExampleRetry?Why
Network failure, timeoutHttpRequestException, TaskCanceledException from a timeout, socket resetYes, if the operation is idempotentThe request may or may not have arrived — only safe when a repeat has no extra effect
HTTP 500, 502, 503, 504Provider overloaded or restartingYes, with backoffUsually temporary
ThrottlingHTTP 429, ThrottlingException, ProvisionedThroughputExceededException, TooManyRequestsExceptionYes, with longer backoffThe service is asking you to slow down; honour Retry-After if present
Client errorHTTP 400, ValidationException, malformed messageNoIt will fail the same way every time; send to a DLQ or reject
AuthorisationHTTP 401/403, AccessDeniedExceptionNo (except once after refreshing a token)Fix configuration, not timing
Business ruleInsufficient funds, account frozen, sanctions hitNoA correct answer, not a failure
ConflictDynamoDB ConditionalCheckFailedExceptionUsually no — re-read and decideSomeone else changed the item; retrying blindly repeats the conflict

Timeouts are the dangerous case

A timeout tells you you stopped waiting, not that the other side failed. The payment may have been accepted. Before you retry anything that moves money, make sure the receiver deduplicates by an idempotency key (see Idempotency in .NET), or query the status first.

Backoff and jitter, worked out

The formula AWS SDKs document for full jitter is:

delay = random(0, 1) × min(cap, base × 2^retry)

retry starts at 0 for the first retry. With base = 200 ms and cap = 20 s:

Retrybase × 2^retryFull-jitter delay range
0200 ms0–200 ms
1400 ms0–400 ms
2800 ms0–800 ms
31.6 s0–1.6 s
612.8 s0–12.8 s
725.6 s → capped0–20 s

Why full jitter (random between zero and the ceiling) rather than "ceiling plus a little noise"? If 1,000 workers all fail at 10:00:00.000, then with no jitter they all retry at exactly 10:00:00.200, again at .600, and so on. Each retry wave is as large as the original spike. With full jitter, their first retries spread evenly across 200 ms. The provider sees a steady trickle instead of synchronised waves. The trade-off is that some clients retry almost immediately. That is acceptable because the aggregate load is what matters.

Variants you will meet:

  • Equal jitter: half the ceiling plus a random half. Guarantees some minimum wait.
  • Decorrelated jitter: each delay is random between base and three times the previous delay. Polly uses a variant of this when UseJitter is on.

All are far better than fixed or no jitter. Pick one, document it, and test it.

The AWS SDK for .NET retry settings

Every SDK client (AmazonSQSClient, AmazonDynamoDBClient and so on) has a config object that inherits from ClientConfig. Two properties control retries:

PropertyValuesEnvironment variable / shared config key
RetryModeRequestRetryMode.Standard, RequestRetryMode.Adaptive, RequestRetryMode.LegacyAWS_RETRY_MODE / retry_mode
MaxErrorRetryNumber of retries (not attempts)AWS_MAX_ATTEMPTS / max_attempts (counts attempts, including the first)

Precedence, from highest to lowest:

  1. Explicit client config in code.
  2. Environment variable.
  3. Shared config file (~/.aws/config).
  4. SDK default.

Standard mode retries transient errors and throttling errors with exponential backoff and full jitter. It also applies the retry quota: a token bucket per client instance. Each retry spends tokens; each success puts some back. When the bucket is empty, the client stops retrying and returns errors immediately. In a widespread outage, your threads therefore fail fast instead of waiting through doomed retries. AWS's cross-SDK reference describes these values for the updated 2026 behaviour:

  • base delays of 50 ms for transient errors and 1,000 ms for throttling;
  • a 20-second cap;
  • a 500-token bucket;
  • a cost of 14 tokens per transient retry and 5 per throttling retry.

Adaptive mode adds a client-side rate limiter. It can delay even the first request after throttling has been seen. AWS recommends it only for clients that target a single resource and expect throttling, for example a batch job writing to one DynamoDB table. It is a poor general default: throttling on one table would slow every call that client makes.

Defaults are changing — set the mode explicitly

At the time of writing, AWS is rolling out updated retry behaviour across SDKs. It is opt-in (AWS_NEW_RETRIES_2026=true) until it becomes the default. The .NET v4 developer guide also states that, without environment settings, the .NET default RetryMode is Legacy. However, the cross-SDK reference lists standard as the default and says legacy is not available for .NET. ⚠ VERIFY which default your SDK version uses (check the tracking issue linked from the AWS retry-behaviour page). Don't depend on the default. Set RetryMode = RequestRetryMode.Standard in code, or AWS_RETRY_MODE=standard in the environment, so behaviour is the same on every machine.

Timeouts belong with retries. In .NET async code, the per-call timeout is the CancellationToken you pass, because the client config's Timeout property does not apply to async calls. A sensible pattern is a per-call token of a few seconds, plus the SDK's own small number of retries inside it.

Retries in AWS messaging services

SQS: the visibility timeout is your retry timer

When a consumer receives a message, SQS hides it for the visibility timeout (30 seconds by default, maximum 12 hours). If the consumer deletes it, processing is done. If not, the message reappears and another receive gets it. That is a retry. Each receive increments the message's ApproximateReceiveCount attribute. When the count passes the redrive policy's maxReceiveCount, SQS moves the message to the DLQ.

With a fixed visibility timeout, every retry happens at the same interval: no backoff. To back off per message, call ChangeMessageVisibility on failure with a delay calculated from ApproximateReceiveCount. The Complete code section does exactly this.

Rules of thumb:

  • Set the queue's visibility timeout comfortably above your normal processing time. For a Lambda event source mapping, AWS recommends at least six times the function timeout.
  • Choose maxReceiveCount together with your backoff. Six receives with backoff of 10 s, 20 s, 40 s, 80 s and 160 s covers about five minutes of provider trouble. For a longer outage, either increase it or pause consumption with a circuit breaker.
  • ChangeMessageVisibility limits: the new value counts from now, but the total cannot exceed 12 hours from the original receive.

Lambda: two different retry worlds

  • Asynchronous invocation (S3, SNS, EventBridge invoking Lambda): Lambda queues the event. On a function error it retries twice, about one minute and then two minutes later. Throttles and system errors are retried for up to six hours. You can lower MaximumRetryAttempts (0–2) and MaximumEventAgeInSeconds (60–21,600). Always set an on-failure destination or DLQ, or failed events vanish.
  • Event source mappings (SQS, Kinesis, DynamoDB Streams, MSK): Lambda polls for you. For SQS, a thrown exception returns the whole batch to the queue. Enable ReportBatchItemFailures and return only the failed message IDs, so successful messages are not reprocessed. For Kinesis and DynamoDB Streams, a failing record blocks its shard until it succeeds or expires. Set MaximumRetryAttempts, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError and an on-failure destination.

SNS

For SQS and Lambda subscribers, SNS retries for a very long time: 100,015 attempts over 23 days. The usual failure is therefore a permission or KMS key problem rather than a slow consumer. For HTTP/S subscribers you can set a delivery policy (backoff function, delays, and maxReceivesPerSecond to protect the endpoint). The total retry time is capped at one hour. Attach a subscription DLQ to keep what SNS gives up on.

EventBridge

On a Classic bus, each rule target has a retry policy. The default is 24 hours and 185 attempts with backoff and jitter. You can lower both (MaximumRetryAttempts, MaximumEventAgeInSeconds). Some failures are not retried at all, for example missing permissions or a target that no longer exists. Those go straight to the target's DLQ if you configured one, and are dropped if you did not. The DLQ must be a standard SQS queue in the same Region.

Step Functions

Retries are declared per state. A good default for a Lambda task:

"Retry": [
  {
    "ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException", "Lambda.TooManyRequestsException"],
    "IntervalSeconds": 1,
    "MaxAttempts": 4,
    "BackoffRate": 2.0,
    "MaxDelaySeconds": 10,
    "JitterStrategy": "FULL"
  },
  {
    "ErrorEquals": ["Harbourline.TransientProviderError"],
    "IntervalSeconds": 5,
    "MaxAttempts": 3,
    "BackoffRate": 2.0,
    "JitterStrategy": "FULL"
  }
]

JitterStrategy defaults to NONE, so turn it on. Business errors (for example Harbourline.InsufficientFunds) should not appear in a retrier. They belong in a Catch that routes to a compensation or rejection state, as the saga lesson shows.

Retry storms and how to prevent them

Retry amplification is multiplicative. If each of n layers makes up to a attempts, a single request can become aⁿ calls at the bottom. With three layers of three attempts, that is 27.

Defences, from most to least important:

  1. One retry owner per hop. Pick the layer closest to the failing dependency, usually the SDK or the queue, and set every other layer to zero automatic retries for that hop.
  2. Retry budgets. The SDK's standard-mode quota does this for AWS calls. For your own calls, Polly's circuit breaker or a simple "retries must stay below 10% of requests" counter does the same job.
  3. Timeouts that shrink on the way down. Give each downstream call a timeout shorter than its caller's, so an outer timeout never fires while inner retries are still running. Use deadline propagation: pass a CancellationToken all the way down.
  4. Circuit breakers. After enough failures, stop calling for a cool-down period and fail fast or pause consumption. Let the queue absorb the backlog.
  5. Load shedding at the edge. API Gateway throttling and usage plans reject excess new work so retries of existing work can complete.
  6. Honour throttling signals. Treat 429 and Retry-After as "slow down", not "try again now".

Queues are the best retry mechanism you have

A message in a queue is durable, costs almost nothing while it waits, and does not hold a thread, a connection or a Lambda invocation. Prefer "fail this attempt quickly and let the queue redeliver later" over "hold the request and retry in memory". In-memory retries are lost when the process restarts, and they tie up resources during exactly the period you need them.

Step-by-step implementation

You build the screening worker from the architecture diagram. It runs locally against a real SQS queue in the sandbox (free tier) or, with no account, against the unit tests only.

Set the SDK retry mode for your profile

Add retry_mode and max_attempts to the learn-sandbox profile so every CLI command and SDK client launched with it behaves the same way:

aws configure set retry_mode standard --profile learn-sandbox
aws configure set max_attempts 3 --profile learn-sandbox

Create the queue and DLQ with a redrive policy

Use the CLI commands in the AWS CLI section. The queue gets a 60-second visibility timeout and maxReceiveCount 6.

Create the worker project

dotnet new worker -n Harbourline.Payments.Screening -f net10.0
cd Harbourline.Payments.Screening
dotnet add package AWSSDK.SQS --version 4.1.0.2
dotnet add package Polly.Core --version 8.9.1

Replace Program.cs and add Backoff.cs, ScreeningWorker.cs and SanctionsClient.cs from Complete code.

Run it and inject failures

Set SANCTIONS_FAILURE_RATE=0.7 to simulate a struggling provider. Send ten messages with the CLI. Watch the logs: each failure schedules the next attempt further away, and after six receives a message lands in the DLQ.

Trip the circuit breaker

Set SANCTIONS_FAILURE_RATE=1.0. After the breaker opens, the worker stops calling the provider and pushes messages back with a long delay. That is your "provider is down" mode.

Add the unit tests

Add an xUnit v3 project and test Backoff.NextDelay (shown in Complete code). Tests need no AWS account.

AWS Console instructions

  1. Console → Amazon SQS → Create queue. Type Standard, name hl-m08-screening-dlq. Keep Encryption → Server-side encryption on (SSE-SQS). Add tags Project=cea-lab, Lab=m08-retries, Environment=learn, Owner=<you>. Create.
  2. Create queue again: name hl-m08-screening. Set Visibility timeout to 60 seconds and Receive message wait time to 20 seconds (long polling). Under Dead-letter queue, enable it, choose hl-m08-screening-dlq and set Maximum receives to 6. Add the same tags. Create.
  3. Open hl-m08-screening → Send and receive messages → send a test body {"paymentId":"PAY-0192d1c2-0000-7000-8000-000000000001"}.
  4. While the worker runs, open the queue's Monitoring tab. Watch Approximate number of messages not visible: that is your in-flight count, including messages waiting for a backed-off retry.
  5. Console → CloudWatch → Alarms → Create alarm on ApproximateNumberOfMessagesVisible for hl-m08-screening-dlq, threshold greater than 0 for 1 datapoint.
  6. For EventBridge targets: Console → Amazon EventBridge → Rules → (rule) → Targets → Edit → Additional settings. Set Maximum age of event and Retry attempts, and choose a Dead-letter queue.
  7. For Lambda async: Console → Lambda → (function) → Configuration → Asynchronous invocation → Edit. Set Maximum age of event and Retry attempts, then add an on-failure destination under Destinations.

AWS CLI

# Common settings for this lesson
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='Project=cea-lab,Lab=m08-retries,Environment=learn,Owner=alex'

# 1. DLQ first, so the main queue can point at it.
DLQ_URL=$(aws sqs create-queue --queue-name hl-m08-screening-dlq \
  --attributes SqsManagedSseEnabled=true,MessageRetentionPeriod=1209600 \
  --tags "$TAGS" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

# 2. Main queue: 60 s visibility, long polling, redrive after 6 receives.
cat > attrs.json <<EOF
{
  "VisibilityTimeout": "60",
  "ReceiveMessageWaitTimeSeconds": "20",
  "SqsManagedSseEnabled": "true",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"6\"}"
}
EOF
Q_URL=$(aws sqs create-queue --queue-name hl-m08-screening \
  --attributes file://attrs.json --tags "$TAGS" --query QueueUrl --output text)

# 3. Send ten test payments to screen.
for i in $(seq 1 10); do
  aws sqs send-message --queue-url "$Q_URL" \
    --message-body "{\"paymentId\":\"PAY-test-$i\",\"creditorName\":\"Example Supplies Ltd\"}"
done

# 4. Manually push one message's next attempt 2 minutes into the future
#    (the worker does this automatically on failure).
RH=$(aws sqs receive-message --queue-url "$Q_URL" --attribute-names ApproximateReceiveCount \
  --query 'Messages[0].ReceiptHandle' --output text)
aws sqs change-message-visibility --queue-url "$Q_URL" --receipt-handle "$RH" --visibility-timeout 120

# 5. Lambda async retry settings for a function (if you have one from earlier lessons).
aws lambda put-function-event-invoke-config --function-name hl-m08-notify \
  --maximum-retry-attempts 1 --maximum-event-age-in-seconds 3600

# 6. EventBridge Classic rule target with a shorter retry policy and a DLQ.
aws events put-targets --rule hl-m08-payment-settled --targets \
  "Id=notify,Arn=arn:aws:lambda:us-east-1:111122223333:function:hl-m08-notify,RetryPolicy={MaximumRetryAttempts=20,MaximumEventAgeInSeconds=7200},DeadLetterConfig={Arn=$DLQ_ARN}"

Notes:

  • MessageRetentionPeriod=1209600 (14 days, the maximum) on the DLQ gives people time to investigate. The source queue's retention should be shorter than or equal to the DLQ's.
  • In step 6 the account ID is a placeholder. For the DLQ, EventBridge also needs permission to send to the queue: add a queue policy allowing events.amazonaws.com to sqs:SendMessage with an aws:SourceArn condition on the rule ARN.

Infrastructure as Code

ScreeningStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Infra;

public sealed class ScreeningStack : Stack
{
    public ScreeningStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var dlq = new Queue(this, "ScreeningDlq", new QueueProps
        {
            QueueName = "hl-m08-screening-dlq",
            Encryption = QueueEncryption.SQS_MANAGED,
            RetentionPeriod = Duration.Days(14),
            EnforceSSL = true
        });

        var queue = new Queue(this, "ScreeningQueue", new QueueProps
        {
            QueueName = "hl-m08-screening",
            Encryption = QueueEncryption.SQS_MANAGED,
            VisibilityTimeout = Duration.Seconds(60),
            ReceiveMessageWaitTime = Duration.Seconds(20),
            EnforceSSL = true,
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 6 }
        });

        // Lambda async retries: one retry, events older than an hour are sent to the failure queue.
        var notify = new Function(this, "Notify", new FunctionProps
        {
            FunctionName = "hl-m08-notify",
            Runtime = Runtime.DOTNET_10,
            Handler = "Harbourline.Notifications::Harbourline.Notifications.Function::Handle",
            Code = Code.FromAsset("../src/Harbourline.Notifications/publish"),
            Timeout = Duration.Seconds(10),
            RetryAttempts = 1,
            MaxEventAge = Duration.Hours(1),
            OnFailure = new Amazon.CDK.AWS.Lambda.Destinations.SqsDestination(dlq)
        });

        // EventBridge Classic rule target with an explicit retry policy and DLQ.
        var rule = new Rule(this, "PaymentSettled", new RuleProps
        {
            RuleName = "hl-m08-payment-settled",
            EventPattern = new EventPattern
            {
                Source = ["harbourline.payments"],
                DetailType = ["PaymentSettled"]
            }
        });
        rule.AddTarget(new LambdaFunction(notify, new LambdaFunctionProps
        {
            RetryAttempts = 20,
            MaxEventAge = Duration.Hours(2),
            DeadLetterQueue = dlq
        }));

        foreach (var (k, v) in new Dictionary<string, string>
                 { ["Project"] = "cea-lab", ["Lab"] = "m08-retries", ["Environment"] = "learn", ["Owner"] = "alex" })
            Amazon.CDK.Tags.Of(this).Add(k, v);

        _ = queue;
    }
}

Runtime.DOTNET_10 requires a recent Amazon.CDK.Lib (2.277.0 in this course). Add Cdklabs.CdkNag checks in the app as described in CONVENTIONS.md.

Complete code

Four files make up the worker. It uses a raw SQS polling loop rather than a framework, so you can see every retry decision; AWS.Messaging for .NET shows the framework version.

Backoff.cs
namespace Harbourline.Payments.Screening;

/// <summary>Exponential backoff with full jitter, in whole seconds (SQS visibility granularity).</summary>
public static class Backoff
{
    public static int NextDelaySeconds(int receiveCount, int baseSeconds, int capSeconds, Random random)
    {
        ArgumentOutOfRangeException.ThrowIfLessThan(receiveCount, 1);
        // receiveCount 1 = first failure → retry index 0.
        var retry = Math.Min(receiveCount - 1, 30);              // avoid overflow in 2^retry
        var ceiling = Math.Min((double)capSeconds, baseSeconds * Math.Pow(2, retry));
        var delay = (int)Math.Ceiling(random.NextDouble() * ceiling);
        return Math.Clamp(delay, 1, Math.Min(capSeconds, 43_200)); // SQS max visibility: 12 h
    }
}
SanctionsClient.cs
namespace Harbourline.Payments.Screening;

public enum ScreeningOutcome { Clear, Hit }

public sealed class TransientProviderException(string message) : Exception(message);
public sealed class PermanentProviderException(string message) : Exception(message);

/// <summary>Fake sanctions provider. Fails transiently at SANCTIONS_FAILURE_RATE.</summary>
public sealed class SanctionsClient
{
    private readonly double _failureRate =
        double.TryParse(Environment.GetEnvironmentVariable("SANCTIONS_FAILURE_RATE"),
            System.Globalization.CultureInfo.InvariantCulture, out var r) ? r : 0.0;

    public async Task<ScreeningOutcome> ScreenAsync(string creditorName, CancellationToken ct)
    {
        await Task.Delay(TimeSpan.FromMilliseconds(Random.Shared.Next(20, 120)), ct);
        if (string.IsNullOrWhiteSpace(creditorName))
            throw new PermanentProviderException("creditorName is required");
        if (Random.Shared.NextDouble() < _failureRate)
            throw new TransientProviderException("503 Service Unavailable (simulated)");
        return creditorName.Contains("BLOCKED", StringComparison.OrdinalIgnoreCase)
            ? ScreeningOutcome.Hit
            : ScreeningOutcome.Clear;
    }
}
ScreeningWorker.cs
using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Polly;
using Polly.CircuitBreaker;

namespace Harbourline.Payments.Screening;

public sealed record ScreeningRequest(string PaymentId, string CreditorName);

public sealed class ScreeningWorker(
    IAmazonSQS sqs,
    SanctionsClient sanctions,
    IConfiguration config,
    ILogger<ScreeningWorker> log) : BackgroundService
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);

    // Circuit breaker only — no retries here. SQS redelivery is the retry owner for this hop.
    private readonly ResiliencePipeline _breaker = new ResiliencePipelineBuilder()
        .AddCircuitBreaker(new CircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            MinimumThroughput = 8,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromSeconds(60),
            ShouldHandle = new PredicateBuilder().Handle<TransientProviderException>().Handle<Polly.Timeout.TimeoutRejectedException>()
        })
        .AddTimeout(TimeSpan.FromSeconds(2))
        .Build();

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var queueUrl = config["Screening:QueueUrl"]
            ?? throw new InvalidOperationException("Screening:QueueUrl is not configured");

        while (!stoppingToken.IsCancellationRequested)
        {
            var response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
            {
                QueueUrl = queueUrl,
                MaxNumberOfMessages = 10,
                WaitTimeSeconds = 20,
                MessageSystemAttributeNames = ["ApproximateReceiveCount"]
            }, stoppingToken);

            // SDK v4: collections are null when empty.
            var messages = response.Messages ?? [];
            await Parallel.ForEachAsync(messages,
                new ParallelOptions { MaxDegreeOfParallelism = 4, CancellationToken = stoppingToken },
                async (m, ct) => await HandleAsync(queueUrl, m, ct));
        }
    }

    private async Task HandleAsync(string queueUrl, Message message, CancellationToken ct)
    {
        var receiveCount = message.Attributes is not null
            && message.Attributes.TryGetValue("ApproximateReceiveCount", out var rc)
            && int.TryParse(rc, out var n) ? n : 1;
        ScreeningRequest? request;
        try
        {
            request = JsonSerializer.Deserialize<ScreeningRequest>(message.Body, Json);
        }
        catch (JsonException)
        {
            request = null;
        }

        if (request is null)
        {
            // Permanent: do not retry. Leave it invisible for the max so it reaches the DLQ quickly
            // only via further receives — or, better, forward it to a "rejected" queue yourself.
            log.LogError("Malformed message {MessageId}; not retrying", message.MessageId);
            await DeferAsync(queueUrl, message, 43_200, ct);
            return;
        }

        try
        {
            var outcome = await _breaker.ExecuteAsync(
                async token => await sanctions.ScreenAsync(request.CreditorName, token), ct);

            log.LogInformation("Payment {PaymentId} screened: {Outcome} (attempt {Attempt})",
                request.PaymentId, outcome, receiveCount);
            // Real code: record the outcome idempotently, then delete.
            await sqs.DeleteMessageAsync(queueUrl, message.ReceiptHandle, ct);
        }
        catch (BrokenCircuitException)
        {
            // Provider is known to be down: don't call it; park the message for a while.
            var delay = Backoff.NextDelaySeconds(receiveCount, baseSeconds: 60, capSeconds: 900, Random.Shared);
            log.LogWarning("Circuit open; deferring {PaymentId} by {Delay}s", request.PaymentId, delay);
            await DeferAsync(queueUrl, message, delay, ct);
        }
        catch (Exception ex) when (ex is TransientProviderException or Polly.Timeout.TimeoutRejectedException)
        {
            var delay = Backoff.NextDelaySeconds(receiveCount, baseSeconds: 10, capSeconds: 300, Random.Shared);
            log.LogWarning("Transient failure for {PaymentId} on attempt {Attempt}; retry in {Delay}s",
                request.PaymentId, receiveCount, delay);
            await DeferAsync(queueUrl, message, delay, ct);
        }
        catch (PermanentProviderException ex)
        {
            log.LogError(ex, "Permanent failure for {PaymentId}; sending to manual review", request.PaymentId);
            // Real code: publish a ScreeningRejected event, then delete. Never loop on permanent errors.
            await sqs.DeleteMessageAsync(queueUrl, message.ReceiptHandle, ct);
        }
    }

    private Task DeferAsync(string queueUrl, Message message, int seconds, CancellationToken ct) =>
        sqs.ChangeMessageVisibilityAsync(new ChangeMessageVisibilityRequest
        {
            QueueUrl = queueUrl,
            ReceiptHandle = message.ReceiptHandle,
            VisibilityTimeout = seconds
        }, ct);
}
Program.cs
using Amazon.Runtime;
using Amazon.SQS;
using Harbourline.Payments.Screening;

var builder = Host.CreateApplicationBuilder(args);

// Explicit retry configuration: standard mode, 2 retries (3 attempts). Credentials come from the
// default chain (AWS_PROFILE=learn-sandbox locally, the task role in ECS). No keys in code.
builder.Services.AddSingleton<IAmazonSQS>(_ => new AmazonSQSClient(new AmazonSQSConfig
{
    RegionEndpoint = Amazon.RegionEndpoint.USEast1,
    RetryMode = RequestRetryMode.Standard,
    MaxErrorRetry = 2
}));
builder.Services.AddSingleton<SanctionsClient>();
builder.Services.AddHostedService<ScreeningWorker>();

builder.Build().Run();
appsettings.json
{
  "Screening": {
    "QueueUrl": "https://sqs.us-east-1.amazonaws.com/111122223333/hl-m08-screening"
  },
  "Logging": { "LogLevel": { "Default": "Information" } }
}
tests/BackoffTests.cs
using Harbourline.Payments.Screening;
using Xunit;

public class BackoffTests
{
    [Theory]
    [InlineData(1, 10)]   // first failure: 0–10 s
    [InlineData(2, 20)]
    [InlineData(4, 80)]
    [InlineData(10, 300)] // capped
    public void Delay_never_exceeds_ceiling(int receiveCount, int expectedCeiling)
    {
        var random = new Random(20261117);
        for (var i = 0; i < 1_000; i++)
        {
            var d = Backoff.NextDelaySeconds(receiveCount, baseSeconds: 10, capSeconds: 300, random);
            Assert.InRange(d, 1, expectedCeiling);
        }
    }

    [Fact]
    public void Delays_are_spread_not_synchronised()
    {
        var random = new Random(20261117);
        var delays = Enumerable.Range(0, 500)
            .Select(_ => Backoff.NextDelaySeconds(3, 10, 300, random))
            .Distinct()
            .Count();
        Assert.True(delays > 20, "full jitter should produce many distinct delays");
    }
}

The test project follows the course pattern: xunit.v3 4.0.3, <OutputType>Exe</OutputType>, Microsoft.Testing.Platform runner in global.json, run with dotnet test --solution <name>.slnx.

Code explained line by line

  1. Backoff.NextDelaySeconds — pure function, so it is easy to unit test. receiveCount - 1 turns SQS's 1-based receive count into a 0-based retry index. Math.Min(..., 30) prevents 2^retry overflowing for poison messages that have been received many times.
  2. random.NextDouble() * ceiling — full jitter: a uniformly random delay between zero and the exponential ceiling. Math.Clamp(delay, 1, ...) keeps at least one second (a zero visibility timeout makes the message visible immediately) and never more than SQS's 12-hour maximum.
  3. TransientProviderException vs PermanentProviderException — the classification table turned into types. The worker never retries permanent errors.
  4. ResiliencePipelineBuilder().AddCircuitBreaker(...) — Polly v8. It opens when at least 50% of at least 8 calls in 30 seconds fail, then stays open for 60 seconds. ShouldHandle lists only transient failures, so a sanctions hit or a validation error never trips the breaker.
  5. .AddTimeout(TimeSpan.FromSeconds(2)) — the provider call gets a hard 2-second budget. It is far shorter than the 60-second visibility timeout, so a slow call can never outlive the message's invisibility (which would cause a duplicate delivery while the first attempt is still running).
  6. There is no AddRetry. This is deliberate. SQS redelivery owns retries for this hop; an in-memory Polly retry here would multiply attempts.
  7. MessageSystemAttributeNames = ["ApproximateReceiveCount"] — asks SQS to include the receive count. In SDK v4 the older AttributeNames property is deprecated in favour of this one ⚠ VERIFY the exact property name against your package version.
  8. response.Messages ?? [] — AWS SDK for .NET v4 returns null instead of empty collections by default. Code written for v3 that assumed an empty list will throw NullReferenceException.
  9. Parallel.ForEachAsync(... MaxDegreeOfParallelism = 4 ...) — bounded concurrency. Unbounded parallelism against a failing provider is a retry storm of its own making.
  10. catch (BrokenCircuitException) — while the breaker is open, the worker does not call the provider. It parks the message for 1–15 minutes (backoff from a 60-second base). The queue absorbs the backlog at almost no cost.
  11. catch (... TransientProviderException or TimeoutRejectedException) — normal transient failures get backoff from a 10-second base, capped at 5 minutes.
  12. DeferAsync → ChangeMessageVisibilityAsync — the per-message backoff. The message stays in the queue and becomes visible after the delay. Its receive count increases on the next receive, so the next delay is longer.
  13. RetryMode = RequestRetryMode.Standard, MaxErrorRetry = 2 — the SDK retries AWS API calls (receive, delete, change visibility) itself. Setting it explicitly avoids the default-mode ambiguity described earlier.
  14. AddSingleton<IAmazonSQS> — one client per process. SDK clients are thread-safe and hold connection pools and the retry-quota token bucket. Creating a client per message would also reset the retry budget each time.

Security considerations

  • Retries must never bypass authorisation. Do not catch AccessDeniedException and retry with a "fallback" credential. Fail and alert.
  • Don't log message bodies on retry. Payment messages contain names and IBANs (personal data under UK GDPR). Log IDs and attempt numbers, as the worker does.
  • DLQs hold personal data too. Encrypt them (SSE-SQS or SSE-KMS), restrict who can read them, and set retention deliberately. The 14-day maximum is usually enough; anything longer must be agreed with data protection.
  • Throttling is also a security control. A client that ignores 429s can look like a denial-of-service attack to a partner and get your IP or API key blocked.
  • Least privilege for the worker: sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility and sqs:GetQueueAttributes on the one queue ARN. Add kms:Decrypt if you use a customer managed key.

⚠ Learning shortcut — not for production

The worker treats a malformed message by hiding it for 12 hours, so it reaches the DLQ only after several more receives. In production, publish malformed messages straight to a "rejected" queue (or the DLQ) with the reason attached, then delete the original. Also persist the screening outcome idempotently before deleting, so a crash between the provider call and the delete doesn't screen and charge for the same payment twice. The breaker state here is per process; with many workers, consider a shared signal such as a CloudWatch alarm that pauses the consumers.

Banking use case

Harbourline's Partner Gateway sends Faster Payments through a scheme connector that enforces strict rate limits and penalises bursts. The platform team set these rules:

  1. Single retry owner. Calls to the connector go through one SQS queue per scheme. The SQS consumer owns retries. API callers get 202 Accepted and a payment ID, and never retry automatically.
  2. Classification is code-reviewed. Every new error code from the connector must be added to a classification table (transient, throttle, permanent, business) before release. Unknown errors default to no retry, manual review. In payments, a wrong "retry" is worse than a delayed answer.
  3. Throttle-aware backoff. Connector 429s use a 30-second base and a circuit breaker that pauses the consumer for the connector's advertised window.
  4. Budgets are monitored. A CloudWatch metric for retries / attempts per dependency raises a warning at 5% and pages at 20%.
  5. Timeouts shrink downward. The API has 10 s, the worker → connector call 3 s, and the SDK calls use the default with a 5-second cancellation token.

Common mistakes

  • Retrying at every layer → the 3 × 3 × 5 explosion from the opening story → choose one retry owner per hop and set the others to zero.
  • Backoff without jitter → synchronised retry waves → always use full or decorrelated jitter; in Step Functions set JitterStrategy: FULL.
  • Retrying 400s and business errors → the same failure five times, then a DLQ full of messages that were never going to work → classify first.
  • Visibility timeout shorter than processing time → a second consumer receives the message while the first is still working → duplicates and double processing → visibility timeout well above the p99 processing time (six times the function timeout for Lambda).
  • Wrapping AWS SDK calls in Polly retries → the SDK's retries times Polly's retries → configure the SDK instead; use Polly for your own HTTP dependencies.
  • Unlimited retries on a stream → one bad record blocks a Kinesis shard for up to the retention period → set MaximumRetryAttempts, BisectBatchOnFunctionError and an on-failure destination.
  • No DLQ on EventBridge targets or SNS subscriptions → events silently dropped after retries → always attach one and alarm on its depth.
  • Creating SDK clients per request → connection-pool churn and a reset retry quota → register clients as singletons.

Troubleshooting

SymptomLikely causeFix
Same message processed twice at about the same timeVisibility timeout shorter than processing timeRaise the visibility timeout; add a per-call timeout shorter than it; make the handler idempotent
DLQ fills with valid messages during a short outagemaxReceiveCount too low for the backoff schedule, or no backoffIncrease maxReceiveCount, add ChangeMessageVisibility backoff, add a circuit breaker
Provider reports "burst" traffic from you after an incidentNo jitter; or many layers retryingAdd full jitter; remove duplicate retry layers
ThrottlingException persists after retriesReal capacity limitRequest a quota increase, add client-side rate limiting, or consider adaptive mode for that single-resource client
Lambda processes the same SQS batch repeatedlyOne failing message causes the whole batch to returnEnable ReportBatchItemFailures and return failed IDs only
EventBridge target never invoked; no retries seenPermission error (not retried)Check the target's resource policy or role; check the DLQ's ERROR_CODE attribute
InvalidParameterValue on ChangeMessageVisibilityTotal invisibility would exceed 12 hours, or receipt handle expiredCap delays; use the latest receipt handle
Retries continue during a full outage, latency climbsLegacy-style retries without a quotaSet RetryMode = Standard; add a circuit breaker

Cost and cleanup

ItemCost at lab volume
SQS requestsFirst 1 million requests per month free (always-free tier ⚠ VERIFY current allowance); this lesson uses a few thousand
Long pollingReduces empty receives, and therefore cost, compared with short polling
Lambda retries (optional)Each retry is a billed invocation; free-tier allowance covers lab use
EventBridge (optional)Custom events are billed per million; lab volume is negligible
CloudWatch alarmSmall monthly charge per alarm beyond the free allowance ⚠ VERIFY

The free alternative is to skip AWS entirely. Run the unit tests and read the worker code; the decisions are the learning.

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-screening --query QueueUrl --output text)"
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-screening-dlq --query QueueUrl --output text)"
aws cloudwatch delete-alarms --alarm-names hl-m08-screening-dlq-depth 2>/dev/null || true

# Confirm nothing tagged for this lesson remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-retries \
  --query "ResourceTagMappingList[].ResourceARN"

The retry_mode and max_attempts lines in your ~/.aws/config profile are worth keeping.

Exercises

  • 🟢 Guided: Build the worker, run it with SANCTIONS_FAILURE_RATE=0.7, send ten messages and record each message's attempt times from the logs. Confirm the gaps grow and are not identical across messages.
  • 🟡 Challenge: Add a second dependency (a fake "account lookup" HTTP API) called through HttpClient with AddStandardResilienceHandler() from Microsoft.Extensions.Http.Resilience. Configure it so that, combined with SQS redelivery, one message can cause at most 6 calls to the lookup API in total.
  • 🔴 Live problem: Harbourline's notification Lambda (SNS → Lambda, async) sends SMS through a partner. During a partner outage last night, customers received the same "payment received" SMS up to three times, and some got none. The on-call engineer wants to "set retries to 10". Write your response and the changes you would make.

Assignment

Write a one-page retry policy for the Harbourline Payments Service and add it to your portfolio repository. Include:

  • the classification table;
  • the retry owner for each hop;
  • the backoff parameters;
  • maxReceiveCount and DLQ ownership;
  • the timeout ladder;
  • the alarms.

Include a Mermaid diagram like the one in this lesson and link the unit-tested Backoff class.

Solutions

Knowledge check

Interview questions

  • "How do you choose retry settings for a service that calls a payment scheme?" — Classify errors first. Retry only transient and throttling errors, and only idempotent operations. Pick one retry owner per hop (usually the queue). Use exponential backoff with full jitter and a cap. Set a timeout ladder so inner timeouts are shorter. Add a circuit breaker and a DLQ with an owner and an alarm. Measure the retry rate.
  • "What is a retry storm and how have you prevented one?" — Layered retries multiply load on a failing dependency. Prevent it with a single retry owner, retry budgets (SDK standard mode or a breaker), jitter, load shedding at the edge, and honouring 429/Retry-After.
  • "Why is a timeout dangerous for payments?" — It is ambiguous: the operation may have succeeded. Retrying without idempotency can duplicate the payment. Use idempotency keys or query the status before retrying.
  • "Standard vs adaptive retry mode in the AWS SDK?" — Standard: backoff with jitter and a retry quota, the right default. Adaptive adds client-side rate limiting that can delay first attempts. It suits a single-resource, throttling-heavy batch client, not a multi-resource service.

Certification practice

Completion checklist

  • I can classify an error as transient, throttling, permanent or business, and say whether to retry it
  • I can compute a full-jitter backoff delay by hand
  • I set retry_mode and max_attempts on my profile and RetryMode explicitly in code
  • I ran the worker with injected failures and saw growing, randomised delays and a DLQ landing
  • I can list each AWS messaging service's default retry behaviour and where failures go
  • I deleted the queues and confirmed nothing tagged Lab=m08-retries remains

Key takeaways

  • Retry only what can succeed next time, and only when repeating it is idempotent.
  • Use exponential backoff with full jitter and a cap; JitterStrategy: FULL in Step Functions is off unless you set it.
  • Have one retry owner per hop. Layered retries multiply into retry storms.
  • The SDK already retries AWS calls. Set RetryMode = Standard explicitly and don't wrap SDK calls in more retries.
  • Queues are the best retry buffer. Back off per message with ChangeMessageVisibility, cap with maxReceiveCount, and always attach a DLQ with an alarm and an owner.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page