Retry strategies: backoff, jitter and avoiding retry storms
Design retries that help rather than hurt: classify errors, use exponential backoff with full jitter, configure AWS SDK for .NET retry modes, line up retries across SQS, Lambda, SNS, EventBridge and Step Functions, and stop layered retries from turning an outage into a retry storm.
Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS 4.1.0.2) · Polly.Core 8.9.1 · Microsoft.Extensions.Http.Resilience 10.11.0 · AWS CLI v2 · Region us-east-1. Verified against official sources on 2026-11-18.
Learning objectives
By the end of this lesson you can:
- Sort a failure into transient, throttling or permanent, and decide whether it should be retried at all.
- Explain exponential backoff with full jitter and calculate the delay for a given attempt.
- Configure the AWS SDK for .NET v4 retry mode and attempt count, in code and through
AWS_RETRY_MODE/AWS_MAX_ATTEMPTS, and explain the retry quota (token bucket). - Describe the built-in retry behaviour of SQS, Lambda (async and event source mappings), SNS, EventBridge and Step Functions, and where each one hands up when it gives up.
- Spot retry amplification (layered retries multiplying load) and fix it with a single retry owner, retry budgets, timeouts and circuit breakers.
- Build an SQS consumer in C# that backs off per message with
ChangeMessageVisibilityinstead of hammering a failing dependency.
Prerequisites
- The SQS, dead-letter-queue, SNS, EventBridge and Step Functions lessons earlier in this module (see the module overview)
- Module 3: .NET on AWS — Worker Services, dependency injection and Lambda basics
- Tools: .NET 10 SDK, AWS CLI v2 with the
learn-sandboxSSO profile
Why this matters in banking
On a Friday afternoon, Harbourline's sanctions-screening provider slows down. Response times go from 80 ms to 9 seconds, then the provider starts returning HTTP 503.
Every layer of the payments platform "helpfully" retries:
- The mobile app retries the payment API three times.
- The Payments API retries the screening call three times, through Polly.
- The AWS SDK inside the screening adapter retries the SQS send three times.
- The Lambda function behind the queue fails, so SQS redelivers the message five times before the dead-letter queue.
One customer tap has become up to 3 × 3 × 5 = 45 screening calls. Across 4,000 payments a minute, the provider now receives roughly 180,000 calls a minute. That is the moment it was least able to cope. The provider's own rate limiter starts rejecting everyone, including Harbourline's other banks' traffic. The incident lasts two hours instead of ten minutes.
Meanwhile some payments were retried after a timeout that had actually succeeded, so a few customers are debited twice.
Retries are the most common resilience tool and the most common cause of self-inflicted outages. A bank must know:
- which layer owns retries;
- how long it waits between attempts;
- when it gives up;
- that every retried operation is idempotent.
The idea in plain language
A retry means "try the same request again because the failure was probably temporary". Three ideas make retries safe:
- Only retry what might succeed next time. A network blip, an HTTP 503 or a throttling error might go away. A validation error, an
AccessDeniedor "insufficient funds" will not. - Wait longer each time, and wait a random amount. Exponential backoff doubles the wait after each failure (100 ms, 200 ms, 400 ms…). Jitter picks a random point within that wait, so thousands of clients that failed at the same moment do not all retry at the same moment.
- Give up, and hand over. After a small number of attempts, or once a time budget is spent, stop. Put the work somewhere safe, such as a dead-letter queue or a failed-payment state, rather than retrying forever.
Analogy: You ring a branch and the line is busy. You do not redial every second, because that ties up the switchboard for everyone. You wait a minute, then a few minutes, then you send an email instead. If every customer redialled on the same second after the same "please call back in one minute" message, the switchboard would stay jammed. Jitter is everyone choosing a slightly different time to call back.
Two more terms you will see:
- Retry storm (retry amplification): retries at several layers multiply. The total load on a failing dependency grows exactly when it can least handle it.
- Retry budget (retry quota): a cap on how many retries a client may make relative to its successful requests. When the budget is spent, the client fails fast instead of piling on.
What problem AWS's retry features solve
AWS builds retries into almost every layer, and you rarely need to write a retry loop around an AWS API call yourself. What you do need is to know each layer's behaviour so they don't stack.
| Layer | Who retries | Default behaviour (checked 2026-11-18) | When it gives up |
|---|---|---|---|
| AWS SDK for .NET (any AWS API call) | The SDK client | Transient and throttling errors retried with backoff and jitter; MaxErrorRetry 2 in standard mode (3 attempts in total). See the SDK section below for the default-mode caveat | Throws the last exception to your code |
| SQS consumer (your poller or Lambda) | SQS redelivers when the visibility timeout expires | Redelivered until maxReceiveCount on the redrive policy, or until the retention period ends | Moves to the DLQ (if configured), otherwise deleted at end of retention |
| Lambda async invoke (S3, SNS, EventBridge → Lambda) | Lambda's internal queue | 2 retries for function errors (about 1 minute, then 2 minutes apart); throttles and system errors retried for up to 6 hours | On-failure destination or DLQ, otherwise discarded |
| Lambda event source mapping for SQS | SQS (through visibility timeout) | Whole batch returns unless you report partial batch failures | Queue's redrive policy |
| Lambda event source mapping for Kinesis / DynamoDB Streams | Lambda | Retries until the record expires by default (MaximumRetryAttempts -1) — blocks the shard | On-failure destination after the limits you set |
| SNS → SQS / Lambda | SNS | AWS-managed endpoints: 100,015 attempts over 23 days | Subscription DLQ, otherwise discarded |
| SNS → HTTP/S | SNS | Customisable delivery policy; total retry time at most 3,600 seconds | Subscription DLQ, otherwise discarded |
| EventBridge rule target (Classic bus) | EventBridge | 24 hours and up to 185 attempts, exponential backoff with jitter | Target DLQ (standard SQS queue), otherwise dropped |
| Step Functions Task | Your Retry block | No retry unless you declare one; defaults when you do: IntervalSeconds 1, MaxAttempts 3, BackoffRate 2.0, JitterStrategy NONE | Catch block, otherwise the execution fails |
Limitations to know:
- Retries cost money and time. Every Lambda retry is a billed invocation. Every Step Functions retry is a billed state transition in Standard workflows. Every SQS redelivery is another
ReceiveMessage. - Not every retry is safe. A retried
SendMessageto a standard queue can create a duplicate message. A retried payment call can create a duplicate payment. Retries assume idempotency, which is covered in Idempotency in .NET. - Retrying hides problems. A service that only works on the third attempt is broken. Measure retry counts and alarm on them.
Microsoft / Azure comparison
| You know (Microsoft/Azure) | AWS equivalent | What's different |
|---|---|---|
Azure SDK RetryOptions (Mode = Exponential, MaxRetries) | AWS SDK for .NET RetryMode and MaxErrorRetry on the client config | AWS also has a cross-SDK retry quota that stops retrying when most calls fail |
Polly / Microsoft.Extensions.Http.Resilience | Same libraries — you still use them for your own HTTP calls | Do not wrap AWS SDK calls in Polly retries as well; the SDK already retries |
Service Bus MaxDeliveryCount | SQS redrive policy maxReceiveCount | SQS retry spacing is controlled by the visibility timeout, not a lock duration |
Service Bus scheduled redelivery / abandon with delay | ChangeMessageVisibility to push a message's next attempt into the future | Per-message, up to 12 hours from the original receive |
Durable Functions RetryOptions | Step Functions Retry with BackoffRate, MaxDelaySeconds, JitterStrategy | Declared in the state machine definition, visible in the console's execution history |
| Event Grid retry policy (24 hours, 30 attempts) | EventBridge target retry policy (24 hours, 185 attempts by default) | Both configurable per subscription/target; both need a dead-letter destination to keep failures |
| Azure Functions host retry policies | Lambda async retry settings and event source mapping settings | Different knobs per event source type |
Architecture
The rule that prevents retry storms is one retry owner per hop. The diagram shows the Harbourline payment-screening path with retries placed deliberately.
Read it like this:
- The app does not auto-retry money movements. If the user resubmits, the same
Idempotency-Keymakes it safe. - The API → SQS hop is an AWS API call, so the SDK owns retries there.
- The worker → provider hop is owned by SQS redelivery. The worker makes one attempt per receive, with a short timeout. On failure it leaves the message in the queue and sets the next visibility timeout using backoff with jitter.
- A circuit breaker in the worker stops calling the provider at all while it is clearly down. Messages simply wait in the queue, which is the cheapest buffer you have.
- After six receives the message goes to the DLQ, and a person looks at it.
Choosing what to retry
Classify before you retry. This table is a starting point for Harbourline services.
| Error | Example | Retry? | Why |
|---|---|---|---|
| Network failure, timeout | HttpRequestException, TaskCanceledException from a timeout, socket reset | Yes, if the operation is idempotent | The request may or may not have arrived — only safe when a repeat has no extra effect |
| HTTP 500, 502, 503, 504 | Provider overloaded or restarting | Yes, with backoff | Usually temporary |
| Throttling | HTTP 429, ThrottlingException, ProvisionedThroughputExceededException, TooManyRequestsException | Yes, with longer backoff | The service is asking you to slow down; honour Retry-After if present |
| Client error | HTTP 400, ValidationException, malformed message | No | It will fail the same way every time; send to a DLQ or reject |
| Authorisation | HTTP 401/403, AccessDeniedException | No (except once after refreshing a token) | Fix configuration, not timing |
| Business rule | Insufficient funds, account frozen, sanctions hit | No | A correct answer, not a failure |
| Conflict | DynamoDB ConditionalCheckFailedException | Usually no — re-read and decide | Someone else changed the item; retrying blindly repeats the conflict |
Timeouts are the dangerous case
A timeout tells you you stopped waiting, not that the other side failed. The payment may have been accepted. Before you retry anything that moves money, make sure the receiver deduplicates by an idempotency key (see Idempotency in .NET), or query the status first.
Backoff and jitter, worked out
The formula AWS SDKs document for full jitter is:
delay = random(0, 1) × min(cap, base × 2^retry)retry starts at 0 for the first retry. With base = 200 ms and cap = 20 s:
| Retry | base × 2^retry | Full-jitter delay range |
|---|---|---|
| 0 | 200 ms | 0–200 ms |
| 1 | 400 ms | 0–400 ms |
| 2 | 800 ms | 0–800 ms |
| 3 | 1.6 s | 0–1.6 s |
| 6 | 12.8 s | 0–12.8 s |
| 7 | 25.6 s → capped | 0–20 s |
Why full jitter (random between zero and the ceiling) rather than "ceiling plus a little noise"? If 1,000 workers all fail at 10:00:00.000, then with no jitter they all retry at exactly 10:00:00.200, again at .600, and so on. Each retry wave is as large as the original spike. With full jitter, their first retries spread evenly across 200 ms. The provider sees a steady trickle instead of synchronised waves. The trade-off is that some clients retry almost immediately. That is acceptable because the aggregate load is what matters.
Variants you will meet:
- Equal jitter: half the ceiling plus a random half. Guarantees some minimum wait.
- Decorrelated jitter: each delay is random between
baseand three times the previous delay. Polly uses a variant of this whenUseJitteris on.
All are far better than fixed or no jitter. Pick one, document it, and test it.
The AWS SDK for .NET retry settings
Every SDK client (AmazonSQSClient, AmazonDynamoDBClient and so on) has a config object that inherits from ClientConfig. Two properties control retries:
| Property | Values | Environment variable / shared config key |
|---|---|---|
RetryMode | RequestRetryMode.Standard, RequestRetryMode.Adaptive, RequestRetryMode.Legacy | AWS_RETRY_MODE / retry_mode |
MaxErrorRetry | Number of retries (not attempts) | AWS_MAX_ATTEMPTS / max_attempts (counts attempts, including the first) |
Precedence, from highest to lowest:
- Explicit client config in code.
- Environment variable.
- Shared config file (
~/.aws/config). - SDK default.
Standard mode retries transient errors and throttling errors with exponential backoff and full jitter. It also applies the retry quota: a token bucket per client instance. Each retry spends tokens; each success puts some back. When the bucket is empty, the client stops retrying and returns errors immediately. In a widespread outage, your threads therefore fail fast instead of waiting through doomed retries. AWS's cross-SDK reference describes these values for the updated 2026 behaviour:
- base delays of 50 ms for transient errors and 1,000 ms for throttling;
- a 20-second cap;
- a 500-token bucket;
- a cost of 14 tokens per transient retry and 5 per throttling retry.
Adaptive mode adds a client-side rate limiter. It can delay even the first request after throttling has been seen. AWS recommends it only for clients that target a single resource and expect throttling, for example a batch job writing to one DynamoDB table. It is a poor general default: throttling on one table would slow every call that client makes.
Defaults are changing — set the mode explicitly
At the time of writing, AWS is rolling out updated retry behaviour across SDKs. It is opt-in (AWS_NEW_RETRIES_2026=true) until it becomes the default. The .NET v4 developer guide also states that, without environment settings, the .NET default RetryMode is Legacy. However, the cross-SDK reference lists standard as the default and says legacy is not available for .NET. ⚠ VERIFY which default your SDK version uses (check the tracking issue linked from the AWS retry-behaviour page). Don't depend on the default. Set RetryMode = RequestRetryMode.Standard in code, or AWS_RETRY_MODE=standard in the environment, so behaviour is the same on every machine.
Timeouts belong with retries. In .NET async code, the per-call timeout is the CancellationToken you pass, because the client config's Timeout property does not apply to async calls. A sensible pattern is a per-call token of a few seconds, plus the SDK's own small number of retries inside it.
Retries in AWS messaging services
SQS: the visibility timeout is your retry timer
When a consumer receives a message, SQS hides it for the visibility timeout (30 seconds by default, maximum 12 hours). If the consumer deletes it, processing is done. If not, the message reappears and another receive gets it. That is a retry. Each receive increments the message's ApproximateReceiveCount attribute. When the count passes the redrive policy's maxReceiveCount, SQS moves the message to the DLQ.
With a fixed visibility timeout, every retry happens at the same interval: no backoff. To back off per message, call ChangeMessageVisibility on failure with a delay calculated from ApproximateReceiveCount. The Complete code section does exactly this.
Rules of thumb:
- Set the queue's visibility timeout comfortably above your normal processing time. For a Lambda event source mapping, AWS recommends at least six times the function timeout.
- Choose
maxReceiveCounttogether with your backoff. Six receives with backoff of 10 s, 20 s, 40 s, 80 s and 160 s covers about five minutes of provider trouble. For a longer outage, either increase it or pause consumption with a circuit breaker. ChangeMessageVisibilitylimits: the new value counts from now, but the total cannot exceed 12 hours from the original receive.
Lambda: two different retry worlds
- Asynchronous invocation (S3, SNS, EventBridge invoking Lambda): Lambda queues the event. On a function error it retries twice, about one minute and then two minutes later. Throttles and system errors are retried for up to six hours. You can lower
MaximumRetryAttempts(0–2) andMaximumEventAgeInSeconds(60–21,600). Always set an on-failure destination or DLQ, or failed events vanish. - Event source mappings (SQS, Kinesis, DynamoDB Streams, MSK): Lambda polls for you. For SQS, a thrown exception returns the whole batch to the queue. Enable
ReportBatchItemFailuresand return only the failed message IDs, so successful messages are not reprocessed. For Kinesis and DynamoDB Streams, a failing record blocks its shard until it succeeds or expires. SetMaximumRetryAttempts,MaximumRecordAgeInSeconds,BisectBatchOnFunctionErrorand an on-failure destination.
SNS
For SQS and Lambda subscribers, SNS retries for a very long time: 100,015 attempts over 23 days. The usual failure is therefore a permission or KMS key problem rather than a slow consumer. For HTTP/S subscribers you can set a delivery policy (backoff function, delays, and maxReceivesPerSecond to protect the endpoint). The total retry time is capped at one hour. Attach a subscription DLQ to keep what SNS gives up on.
EventBridge
On a Classic bus, each rule target has a retry policy. The default is 24 hours and 185 attempts with backoff and jitter. You can lower both (MaximumRetryAttempts, MaximumEventAgeInSeconds). Some failures are not retried at all, for example missing permissions or a target that no longer exists. Those go straight to the target's DLQ if you configured one, and are dropped if you did not. The DLQ must be a standard SQS queue in the same Region.
Step Functions
Retries are declared per state. A good default for a Lambda task:
"Retry": [
{
"ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException", "Lambda.TooManyRequestsException"],
"IntervalSeconds": 1,
"MaxAttempts": 4,
"BackoffRate": 2.0,
"MaxDelaySeconds": 10,
"JitterStrategy": "FULL"
},
{
"ErrorEquals": ["Harbourline.TransientProviderError"],
"IntervalSeconds": 5,
"MaxAttempts": 3,
"BackoffRate": 2.0,
"JitterStrategy": "FULL"
}
]JitterStrategy defaults to NONE, so turn it on. Business errors (for example Harbourline.InsufficientFunds) should not appear in a retrier. They belong in a Catch that routes to a compensation or rejection state, as the saga lesson shows.
Retry storms and how to prevent them
Retry amplification is multiplicative. If each of n layers makes up to a attempts, a single request can become aⁿ calls at the bottom. With three layers of three attempts, that is 27.
Defences, from most to least important:
- One retry owner per hop. Pick the layer closest to the failing dependency, usually the SDK or the queue, and set every other layer to zero automatic retries for that hop.
- Retry budgets. The SDK's standard-mode quota does this for AWS calls. For your own calls, Polly's circuit breaker or a simple "retries must stay below 10% of requests" counter does the same job.
- Timeouts that shrink on the way down. Give each downstream call a timeout shorter than its caller's, so an outer timeout never fires while inner retries are still running. Use deadline propagation: pass a
CancellationTokenall the way down. - Circuit breakers. After enough failures, stop calling for a cool-down period and fail fast or pause consumption. Let the queue absorb the backlog.
- Load shedding at the edge. API Gateway throttling and usage plans reject excess new work so retries of existing work can complete.
- Honour throttling signals. Treat 429 and
Retry-Afteras "slow down", not "try again now".
Queues are the best retry mechanism you have
A message in a queue is durable, costs almost nothing while it waits, and does not hold a thread, a connection or a Lambda invocation. Prefer "fail this attempt quickly and let the queue redeliver later" over "hold the request and retry in memory". In-memory retries are lost when the process restarts, and they tie up resources during exactly the period you need them.
Step-by-step implementation
You build the screening worker from the architecture diagram. It runs locally against a real SQS queue in the sandbox (free tier) or, with no account, against the unit tests only.
Set the SDK retry mode for your profile
Add retry_mode and max_attempts to the learn-sandbox profile so every CLI command and SDK client launched with it behaves the same way:
aws configure set retry_mode standard --profile learn-sandbox
aws configure set max_attempts 3 --profile learn-sandboxCreate the queue and DLQ with a redrive policy
Use the CLI commands in the AWS CLI section. The queue gets a 60-second visibility timeout and maxReceiveCount 6.
Create the worker project
dotnet new worker -n Harbourline.Payments.Screening -f net10.0
cd Harbourline.Payments.Screening
dotnet add package AWSSDK.SQS --version 4.1.0.2
dotnet add package Polly.Core --version 8.9.1Replace Program.cs and add Backoff.cs, ScreeningWorker.cs and SanctionsClient.cs from Complete code.
Run it and inject failures
Set SANCTIONS_FAILURE_RATE=0.7 to simulate a struggling provider. Send ten messages with the CLI. Watch the logs: each failure schedules the next attempt further away, and after six receives a message lands in the DLQ.
Trip the circuit breaker
Set SANCTIONS_FAILURE_RATE=1.0. After the breaker opens, the worker stops calling the provider and pushes messages back with a long delay. That is your "provider is down" mode.
Add the unit tests
Add an xUnit v3 project and test Backoff.NextDelay (shown in Complete code). Tests need no AWS account.
AWS Console instructions
- Console → Amazon SQS → Create queue. Type Standard, name
hl-m08-screening-dlq. Keep Encryption → Server-side encryption on (SSE-SQS). Add tagsProject=cea-lab,Lab=m08-retries,Environment=learn,Owner=<you>. Create. - Create queue again: name
hl-m08-screening. Set Visibility timeout to 60 seconds and Receive message wait time to 20 seconds (long polling). Under Dead-letter queue, enable it, choosehl-m08-screening-dlqand set Maximum receives to 6. Add the same tags. Create. - Open
hl-m08-screening→ Send and receive messages → send a test body{"paymentId":"PAY-0192d1c2-0000-7000-8000-000000000001"}. - While the worker runs, open the queue's Monitoring tab. Watch Approximate number of messages not visible: that is your in-flight count, including messages waiting for a backed-off retry.
- Console → CloudWatch → Alarms → Create alarm on
ApproximateNumberOfMessagesVisibleforhl-m08-screening-dlq, threshold greater than 0 for 1 datapoint. - For EventBridge targets: Console → Amazon EventBridge → Rules → (rule) → Targets → Edit → Additional settings. Set Maximum age of event and Retry attempts, and choose a Dead-letter queue.
- For Lambda async: Console → Lambda → (function) → Configuration → Asynchronous invocation → Edit. Set Maximum age of event and Retry attempts, then add an on-failure destination under Destinations.
AWS CLI
# Common settings for this lesson
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='Project=cea-lab,Lab=m08-retries,Environment=learn,Owner=alex'
# 1. DLQ first, so the main queue can point at it.
DLQ_URL=$(aws sqs create-queue --queue-name hl-m08-screening-dlq \
--attributes SqsManagedSseEnabled=true,MessageRetentionPeriod=1209600 \
--tags "$TAGS" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
--attribute-names QueueArn --query Attributes.QueueArn --output text)
# 2. Main queue: 60 s visibility, long polling, redrive after 6 receives.
cat > attrs.json <<EOF
{
"VisibilityTimeout": "60",
"ReceiveMessageWaitTimeSeconds": "20",
"SqsManagedSseEnabled": "true",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"6\"}"
}
EOF
Q_URL=$(aws sqs create-queue --queue-name hl-m08-screening \
--attributes file://attrs.json --tags "$TAGS" --query QueueUrl --output text)
# 3. Send ten test payments to screen.
for i in $(seq 1 10); do
aws sqs send-message --queue-url "$Q_URL" \
--message-body "{\"paymentId\":\"PAY-test-$i\",\"creditorName\":\"Example Supplies Ltd\"}"
done
# 4. Manually push one message's next attempt 2 minutes into the future
# (the worker does this automatically on failure).
RH=$(aws sqs receive-message --queue-url "$Q_URL" --attribute-names ApproximateReceiveCount \
--query 'Messages[0].ReceiptHandle' --output text)
aws sqs change-message-visibility --queue-url "$Q_URL" --receipt-handle "$RH" --visibility-timeout 120
# 5. Lambda async retry settings for a function (if you have one from earlier lessons).
aws lambda put-function-event-invoke-config --function-name hl-m08-notify \
--maximum-retry-attempts 1 --maximum-event-age-in-seconds 3600
# 6. EventBridge Classic rule target with a shorter retry policy and a DLQ.
aws events put-targets --rule hl-m08-payment-settled --targets \
"Id=notify,Arn=arn:aws:lambda:us-east-1:111122223333:function:hl-m08-notify,RetryPolicy={MaximumRetryAttempts=20,MaximumEventAgeInSeconds=7200},DeadLetterConfig={Arn=$DLQ_ARN}"Notes:
MessageRetentionPeriod=1209600(14 days, the maximum) on the DLQ gives people time to investigate. The source queue's retention should be shorter than or equal to the DLQ's.- In step 6 the account ID is a placeholder. For the DLQ, EventBridge also needs permission to send to the queue: add a queue policy allowing
events.amazonaws.comtosqs:SendMessagewith anaws:SourceArncondition on the rule ARN.
Infrastructure as Code
using Amazon.CDK;
using Amazon.CDK.AWS.Events;
using Amazon.CDK.AWS.Events.Targets;
using Amazon.CDK.AWS.Lambda;
using Amazon.CDK.AWS.SQS;
using Constructs;
namespace Harbourline.Infra;
public sealed class ScreeningStack : Stack
{
public ScreeningStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
{
var dlq = new Queue(this, "ScreeningDlq", new QueueProps
{
QueueName = "hl-m08-screening-dlq",
Encryption = QueueEncryption.SQS_MANAGED,
RetentionPeriod = Duration.Days(14),
EnforceSSL = true
});
var queue = new Queue(this, "ScreeningQueue", new QueueProps
{
QueueName = "hl-m08-screening",
Encryption = QueueEncryption.SQS_MANAGED,
VisibilityTimeout = Duration.Seconds(60),
ReceiveMessageWaitTime = Duration.Seconds(20),
EnforceSSL = true,
DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 6 }
});
// Lambda async retries: one retry, events older than an hour are sent to the failure queue.
var notify = new Function(this, "Notify", new FunctionProps
{
FunctionName = "hl-m08-notify",
Runtime = Runtime.DOTNET_10,
Handler = "Harbourline.Notifications::Harbourline.Notifications.Function::Handle",
Code = Code.FromAsset("../src/Harbourline.Notifications/publish"),
Timeout = Duration.Seconds(10),
RetryAttempts = 1,
MaxEventAge = Duration.Hours(1),
OnFailure = new Amazon.CDK.AWS.Lambda.Destinations.SqsDestination(dlq)
});
// EventBridge Classic rule target with an explicit retry policy and DLQ.
var rule = new Rule(this, "PaymentSettled", new RuleProps
{
RuleName = "hl-m08-payment-settled",
EventPattern = new EventPattern
{
Source = ["harbourline.payments"],
DetailType = ["PaymentSettled"]
}
});
rule.AddTarget(new LambdaFunction(notify, new LambdaFunctionProps
{
RetryAttempts = 20,
MaxEventAge = Duration.Hours(2),
DeadLetterQueue = dlq
}));
foreach (var (k, v) in new Dictionary<string, string>
{ ["Project"] = "cea-lab", ["Lab"] = "m08-retries", ["Environment"] = "learn", ["Owner"] = "alex" })
Amazon.CDK.Tags.Of(this).Add(k, v);
_ = queue;
}
}Runtime.DOTNET_10 requires a recent Amazon.CDK.Lib (2.277.0 in this course). Add Cdklabs.CdkNag checks in the app as described in CONVENTIONS.md.
Complete code
Four files make up the worker. It uses a raw SQS polling loop rather than a framework, so you can see every retry decision; AWS.Messaging for .NET shows the framework version.
namespace Harbourline.Payments.Screening;
/// <summary>Exponential backoff with full jitter, in whole seconds (SQS visibility granularity).</summary>
public static class Backoff
{
public static int NextDelaySeconds(int receiveCount, int baseSeconds, int capSeconds, Random random)
{
ArgumentOutOfRangeException.ThrowIfLessThan(receiveCount, 1);
// receiveCount 1 = first failure → retry index 0.
var retry = Math.Min(receiveCount - 1, 30); // avoid overflow in 2^retry
var ceiling = Math.Min((double)capSeconds, baseSeconds * Math.Pow(2, retry));
var delay = (int)Math.Ceiling(random.NextDouble() * ceiling);
return Math.Clamp(delay, 1, Math.Min(capSeconds, 43_200)); // SQS max visibility: 12 h
}
}namespace Harbourline.Payments.Screening;
public enum ScreeningOutcome { Clear, Hit }
public sealed class TransientProviderException(string message) : Exception(message);
public sealed class PermanentProviderException(string message) : Exception(message);
/// <summary>Fake sanctions provider. Fails transiently at SANCTIONS_FAILURE_RATE.</summary>
public sealed class SanctionsClient
{
private readonly double _failureRate =
double.TryParse(Environment.GetEnvironmentVariable("SANCTIONS_FAILURE_RATE"),
System.Globalization.CultureInfo.InvariantCulture, out var r) ? r : 0.0;
public async Task<ScreeningOutcome> ScreenAsync(string creditorName, CancellationToken ct)
{
await Task.Delay(TimeSpan.FromMilliseconds(Random.Shared.Next(20, 120)), ct);
if (string.IsNullOrWhiteSpace(creditorName))
throw new PermanentProviderException("creditorName is required");
if (Random.Shared.NextDouble() < _failureRate)
throw new TransientProviderException("503 Service Unavailable (simulated)");
return creditorName.Contains("BLOCKED", StringComparison.OrdinalIgnoreCase)
? ScreeningOutcome.Hit
: ScreeningOutcome.Clear;
}
}using System.Text.Json;
using Amazon.SQS;
using Amazon.SQS.Model;
using Polly;
using Polly.CircuitBreaker;
namespace Harbourline.Payments.Screening;
public sealed record ScreeningRequest(string PaymentId, string CreditorName);
public sealed class ScreeningWorker(
IAmazonSQS sqs,
SanctionsClient sanctions,
IConfiguration config,
ILogger<ScreeningWorker> log) : BackgroundService
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
// Circuit breaker only — no retries here. SQS redelivery is the retry owner for this hop.
private readonly ResiliencePipeline _breaker = new ResiliencePipelineBuilder()
.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
MinimumThroughput = 8,
SamplingDuration = TimeSpan.FromSeconds(30),
BreakDuration = TimeSpan.FromSeconds(60),
ShouldHandle = new PredicateBuilder().Handle<TransientProviderException>().Handle<Polly.Timeout.TimeoutRejectedException>()
})
.AddTimeout(TimeSpan.FromSeconds(2))
.Build();
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var queueUrl = config["Screening:QueueUrl"]
?? throw new InvalidOperationException("Screening:QueueUrl is not configured");
while (!stoppingToken.IsCancellationRequested)
{
var response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
{
QueueUrl = queueUrl,
MaxNumberOfMessages = 10,
WaitTimeSeconds = 20,
MessageSystemAttributeNames = ["ApproximateReceiveCount"]
}, stoppingToken);
// SDK v4: collections are null when empty.
var messages = response.Messages ?? [];
await Parallel.ForEachAsync(messages,
new ParallelOptions { MaxDegreeOfParallelism = 4, CancellationToken = stoppingToken },
async (m, ct) => await HandleAsync(queueUrl, m, ct));
}
}
private async Task HandleAsync(string queueUrl, Message message, CancellationToken ct)
{
var receiveCount = message.Attributes is not null
&& message.Attributes.TryGetValue("ApproximateReceiveCount", out var rc)
&& int.TryParse(rc, out var n) ? n : 1;
ScreeningRequest? request;
try
{
request = JsonSerializer.Deserialize<ScreeningRequest>(message.Body, Json);
}
catch (JsonException)
{
request = null;
}
if (request is null)
{
// Permanent: do not retry. Leave it invisible for the max so it reaches the DLQ quickly
// only via further receives — or, better, forward it to a "rejected" queue yourself.
log.LogError("Malformed message {MessageId}; not retrying", message.MessageId);
await DeferAsync(queueUrl, message, 43_200, ct);
return;
}
try
{
var outcome = await _breaker.ExecuteAsync(
async token => await sanctions.ScreenAsync(request.CreditorName, token), ct);
log.LogInformation("Payment {PaymentId} screened: {Outcome} (attempt {Attempt})",
request.PaymentId, outcome, receiveCount);
// Real code: record the outcome idempotently, then delete.
await sqs.DeleteMessageAsync(queueUrl, message.ReceiptHandle, ct);
}
catch (BrokenCircuitException)
{
// Provider is known to be down: don't call it; park the message for a while.
var delay = Backoff.NextDelaySeconds(receiveCount, baseSeconds: 60, capSeconds: 900, Random.Shared);
log.LogWarning("Circuit open; deferring {PaymentId} by {Delay}s", request.PaymentId, delay);
await DeferAsync(queueUrl, message, delay, ct);
}
catch (Exception ex) when (ex is TransientProviderException or Polly.Timeout.TimeoutRejectedException)
{
var delay = Backoff.NextDelaySeconds(receiveCount, baseSeconds: 10, capSeconds: 300, Random.Shared);
log.LogWarning("Transient failure for {PaymentId} on attempt {Attempt}; retry in {Delay}s",
request.PaymentId, receiveCount, delay);
await DeferAsync(queueUrl, message, delay, ct);
}
catch (PermanentProviderException ex)
{
log.LogError(ex, "Permanent failure for {PaymentId}; sending to manual review", request.PaymentId);
// Real code: publish a ScreeningRejected event, then delete. Never loop on permanent errors.
await sqs.DeleteMessageAsync(queueUrl, message.ReceiptHandle, ct);
}
}
private Task DeferAsync(string queueUrl, Message message, int seconds, CancellationToken ct) =>
sqs.ChangeMessageVisibilityAsync(new ChangeMessageVisibilityRequest
{
QueueUrl = queueUrl,
ReceiptHandle = message.ReceiptHandle,
VisibilityTimeout = seconds
}, ct);
}using Amazon.Runtime;
using Amazon.SQS;
using Harbourline.Payments.Screening;
var builder = Host.CreateApplicationBuilder(args);
// Explicit retry configuration: standard mode, 2 retries (3 attempts). Credentials come from the
// default chain (AWS_PROFILE=learn-sandbox locally, the task role in ECS). No keys in code.
builder.Services.AddSingleton<IAmazonSQS>(_ => new AmazonSQSClient(new AmazonSQSConfig
{
RegionEndpoint = Amazon.RegionEndpoint.USEast1,
RetryMode = RequestRetryMode.Standard,
MaxErrorRetry = 2
}));
builder.Services.AddSingleton<SanctionsClient>();
builder.Services.AddHostedService<ScreeningWorker>();
builder.Build().Run();{
"Screening": {
"QueueUrl": "https://sqs.us-east-1.amazonaws.com/111122223333/hl-m08-screening"
},
"Logging": { "LogLevel": { "Default": "Information" } }
}using Harbourline.Payments.Screening;
using Xunit;
public class BackoffTests
{
[Theory]
[InlineData(1, 10)] // first failure: 0–10 s
[InlineData(2, 20)]
[InlineData(4, 80)]
[InlineData(10, 300)] // capped
public void Delay_never_exceeds_ceiling(int receiveCount, int expectedCeiling)
{
var random = new Random(20261117);
for (var i = 0; i < 1_000; i++)
{
var d = Backoff.NextDelaySeconds(receiveCount, baseSeconds: 10, capSeconds: 300, random);
Assert.InRange(d, 1, expectedCeiling);
}
}
[Fact]
public void Delays_are_spread_not_synchronised()
{
var random = new Random(20261117);
var delays = Enumerable.Range(0, 500)
.Select(_ => Backoff.NextDelaySeconds(3, 10, 300, random))
.Distinct()
.Count();
Assert.True(delays > 20, "full jitter should produce many distinct delays");
}
}The test project follows the course pattern: xunit.v3 4.0.3, <OutputType>Exe</OutputType>, Microsoft.Testing.Platform runner in global.json, run with dotnet test --solution <name>.slnx.
Code explained line by line
Backoff.NextDelaySeconds— pure function, so it is easy to unit test.receiveCount - 1turns SQS's 1-based receive count into a 0-based retry index.Math.Min(..., 30)prevents2^retryoverflowing for poison messages that have been received many times.random.NextDouble() * ceiling— full jitter: a uniformly random delay between zero and the exponential ceiling.Math.Clamp(delay, 1, ...)keeps at least one second (a zero visibility timeout makes the message visible immediately) and never more than SQS's 12-hour maximum.TransientProviderExceptionvsPermanentProviderException— the classification table turned into types. The worker never retries permanent errors.ResiliencePipelineBuilder().AddCircuitBreaker(...)— Polly v8. It opens when at least 50% of at least 8 calls in 30 seconds fail, then stays open for 60 seconds.ShouldHandlelists only transient failures, so a sanctions hit or a validation error never trips the breaker..AddTimeout(TimeSpan.FromSeconds(2))— the provider call gets a hard 2-second budget. It is far shorter than the 60-second visibility timeout, so a slow call can never outlive the message's invisibility (which would cause a duplicate delivery while the first attempt is still running).- There is no
AddRetry. This is deliberate. SQS redelivery owns retries for this hop; an in-memory Polly retry here would multiply attempts. MessageSystemAttributeNames = ["ApproximateReceiveCount"]— asks SQS to include the receive count. In SDK v4 the olderAttributeNamesproperty is deprecated in favour of this one ⚠ VERIFY the exact property name against your package version.response.Messages ?? []— AWS SDK for .NET v4 returnsnullinstead of empty collections by default. Code written for v3 that assumed an empty list will throwNullReferenceException.Parallel.ForEachAsync(... MaxDegreeOfParallelism = 4 ...)— bounded concurrency. Unbounded parallelism against a failing provider is a retry storm of its own making.catch (BrokenCircuitException)— while the breaker is open, the worker does not call the provider. It parks the message for 1–15 minutes (backoff from a 60-second base). The queue absorbs the backlog at almost no cost.catch (... TransientProviderException or TimeoutRejectedException)— normal transient failures get backoff from a 10-second base, capped at 5 minutes.DeferAsync→ChangeMessageVisibilityAsync— the per-message backoff. The message stays in the queue and becomes visible after the delay. Its receive count increases on the next receive, so the next delay is longer.RetryMode = RequestRetryMode.Standard, MaxErrorRetry = 2— the SDK retries AWS API calls (receive, delete, change visibility) itself. Setting it explicitly avoids the default-mode ambiguity described earlier.AddSingleton<IAmazonSQS>— one client per process. SDK clients are thread-safe and hold connection pools and the retry-quota token bucket. Creating a client per message would also reset the retry budget each time.
Security considerations
- Retries must never bypass authorisation. Do not catch
AccessDeniedExceptionand retry with a "fallback" credential. Fail and alert. - Don't log message bodies on retry. Payment messages contain names and IBANs (personal data under UK GDPR). Log IDs and attempt numbers, as the worker does.
- DLQs hold personal data too. Encrypt them (SSE-SQS or SSE-KMS), restrict who can read them, and set retention deliberately. The 14-day maximum is usually enough; anything longer must be agreed with data protection.
- Throttling is also a security control. A client that ignores 429s can look like a denial-of-service attack to a partner and get your IP or API key blocked.
- Least privilege for the worker:
sqs:ReceiveMessage,sqs:DeleteMessage,sqs:ChangeMessageVisibilityandsqs:GetQueueAttributeson the one queue ARN. Addkms:Decryptif you use a customer managed key.
⚠ Learning shortcut — not for production
The worker treats a malformed message by hiding it for 12 hours, so it reaches the DLQ only after several more receives. In production, publish malformed messages straight to a "rejected" queue (or the DLQ) with the reason attached, then delete the original. Also persist the screening outcome idempotently before deleting, so a crash between the provider call and the delete doesn't screen and charge for the same payment twice. The breaker state here is per process; with many workers, consider a shared signal such as a CloudWatch alarm that pauses the consumers.
Banking use case
Harbourline's Partner Gateway sends Faster Payments through a scheme connector that enforces strict rate limits and penalises bursts. The platform team set these rules:
- Single retry owner. Calls to the connector go through one SQS queue per scheme. The SQS consumer owns retries. API callers get
202 Acceptedand a payment ID, and never retry automatically. - Classification is code-reviewed. Every new error code from the connector must be added to a classification table (transient, throttle, permanent, business) before release. Unknown errors default to no retry, manual review. In payments, a wrong "retry" is worse than a delayed answer.
- Throttle-aware backoff. Connector 429s use a 30-second base and a circuit breaker that pauses the consumer for the connector's advertised window.
- Budgets are monitored. A CloudWatch metric for
retries / attemptsper dependency raises a warning at 5% and pages at 20%. - Timeouts shrink downward. The API has 10 s, the worker → connector call 3 s, and the SDK calls use the default with a 5-second cancellation token.
Common mistakes
- Retrying at every layer → the 3 × 3 × 5 explosion from the opening story → choose one retry owner per hop and set the others to zero.
- Backoff without jitter → synchronised retry waves → always use full or decorrelated jitter; in Step Functions set
JitterStrategy: FULL. - Retrying 400s and business errors → the same failure five times, then a DLQ full of messages that were never going to work → classify first.
- Visibility timeout shorter than processing time → a second consumer receives the message while the first is still working → duplicates and double processing → visibility timeout well above the p99 processing time (six times the function timeout for Lambda).
- Wrapping AWS SDK calls in Polly retries → the SDK's retries times Polly's retries → configure the SDK instead; use Polly for your own HTTP dependencies.
- Unlimited retries on a stream → one bad record blocks a Kinesis shard for up to the retention period → set
MaximumRetryAttempts,BisectBatchOnFunctionErrorand an on-failure destination. - No DLQ on EventBridge targets or SNS subscriptions → events silently dropped after retries → always attach one and alarm on its depth.
- Creating SDK clients per request → connection-pool churn and a reset retry quota → register clients as singletons.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Same message processed twice at about the same time | Visibility timeout shorter than processing time | Raise the visibility timeout; add a per-call timeout shorter than it; make the handler idempotent |
| DLQ fills with valid messages during a short outage | maxReceiveCount too low for the backoff schedule, or no backoff | Increase maxReceiveCount, add ChangeMessageVisibility backoff, add a circuit breaker |
| Provider reports "burst" traffic from you after an incident | No jitter; or many layers retrying | Add full jitter; remove duplicate retry layers |
ThrottlingException persists after retries | Real capacity limit | Request a quota increase, add client-side rate limiting, or consider adaptive mode for that single-resource client |
| Lambda processes the same SQS batch repeatedly | One failing message causes the whole batch to return | Enable ReportBatchItemFailures and return failed IDs only |
| EventBridge target never invoked; no retries seen | Permission error (not retried) | Check the target's resource policy or role; check the DLQ's ERROR_CODE attribute |
InvalidParameterValue on ChangeMessageVisibility | Total invisibility would exceed 12 hours, or receipt handle expired | Cap delays; use the latest receipt handle |
| Retries continue during a full outage, latency climbs | Legacy-style retries without a quota | Set RetryMode = Standard; add a circuit breaker |
Cost and cleanup
| Item | Cost at lab volume |
|---|---|
| SQS requests | First 1 million requests per month free (always-free tier ⚠ VERIFY current allowance); this lesson uses a few thousand |
| Long polling | Reduces empty receives, and therefore cost, compared with short polling |
| Lambda retries (optional) | Each retry is a billed invocation; free-tier allowance covers lab use |
| EventBridge (optional) | Custom events are billed per million; lab volume is negligible |
| CloudWatch alarm | Small monthly charge per alarm beyond the free allowance ⚠ VERIFY |
The free alternative is to skip AWS entirely. Run the unit tests and read the worker code; the decisions are the learning.
export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-screening --query QueueUrl --output text)"
aws sqs delete-queue --queue-url "$(aws sqs get-queue-url --queue-name hl-m08-screening-dlq --query QueueUrl --output text)"
aws cloudwatch delete-alarms --alarm-names hl-m08-screening-dlq-depth 2>/dev/null || true
# Confirm nothing tagged for this lesson remains.
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m08-retries \
--query "ResourceTagMappingList[].ResourceARN"The retry_mode and max_attempts lines in your ~/.aws/config profile are worth keeping.
Exercises
- 🟢 Guided: Build the worker, run it with
SANCTIONS_FAILURE_RATE=0.7, send ten messages and record each message's attempt times from the logs. Confirm the gaps grow and are not identical across messages. - 🟡 Challenge: Add a second dependency (a fake "account lookup" HTTP API) called through
HttpClientwithAddStandardResilienceHandler()fromMicrosoft.Extensions.Http.Resilience. Configure it so that, combined with SQS redelivery, one message can cause at most 6 calls to the lookup API in total. - 🔴 Live problem: Harbourline's notification Lambda (SNS → Lambda, async) sends SMS through a partner. During a partner outage last night, customers received the same "payment received" SMS up to three times, and some got none. The on-call engineer wants to "set retries to 10". Write your response and the changes you would make.
Assignment
Write a one-page retry policy for the Harbourline Payments Service and add it to your portfolio repository. Include:
- the classification table;
- the retry owner for each hop;
- the backoff parameters;
maxReceiveCountand DLQ ownership;- the timeout ladder;
- the alarms.
Include a Mermaid diagram like the one in this lesson and link the unit-tested Backoff class.
Solutions
Typical log lines for one message:
Transient failure for PAY-test-3 on attempt 1; retry in 7s
Transient failure for PAY-test-3 on attempt 2; retry in 14s
Transient failure for PAY-test-3 on attempt 3; retry in 33s
Payment PAY-test-3 screened: Clear (attempt 4)Another message might show 2 s, 19 s, 12 s. The ceilings (10, 20, 40 seconds) grow, but each actual delay is random below its ceiling, and no two messages share a schedule.
Why: This is full jitter working. If you see identical delays across messages, the Random instance is being re-created with the same seed. Use Random.Shared in production code.
builder.Services.AddHttpClient("account-lookup", c =>
{
c.BaseAddress = new Uri("https://accounts.internal.example.net/");
})
.AddStandardResilienceHandler(o =>
{
o.Retry.MaxRetryAttempts = 0; // SQS owns retries for this hop
o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(5);
o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
});With maxReceiveCount = 6 and zero HTTP retries, one message causes at most 6 lookup calls. If you prefer one fast HTTP retry for connection resets (MaxRetryAttempts = 1), you must reduce maxReceiveCount to 3: 3 receives × 2 attempts = 6.
Why: The budget is the product of the layers. Write the product down and design to it.
Don't set retries to 10. There are two separate problems.
- Triple SMS: Lambda async invocation retries a failed function twice by default. The function sent the SMS, then failed afterwards (for example, it timed out while waiting for the partner's response), so the retries sent the SMS again. Fix: make the send idempotent. Record
notificationIdin a DynamoDB idempotency table with a conditional write before calling the partner, and use the partner's idempotency key if it offers one. - Missing SMS: after the retries and the maximum event age, events were discarded because no on-failure destination was configured. Fix: add an on-failure destination (SQS) and an alarm on its depth.
- Better design: put an SQS queue between SNS and the sender (SNS → SQS → Lambda event source mapping). The queue becomes the single retry owner, with backoff via visibility and a DLQ with redrive. Add a circuit breaker for the partner and a shorter function timeout than the visibility timeout.
Why: More retries would have produced more duplicate SMS and still lost messages once the event age expired. Duplicates need idempotency; loss needs a destination.
Knowledge check
Answer: Backoff increases the ceiling of the wait after each failure, so a struggling dependency gets more breathing room. Jitter randomises the actual wait below that ceiling, so clients that failed together do not retry together. Backoff alone still produces synchronised waves; jitter alone doesn't slow down enough.
Answer: 3 gateway attempts × 4 receives × 3 Polly attempts = 36 if every layer fires. This is retry amplification. Remove all but one retry owner per hop.
Answer: Retrying during a widespread failure. Retries spend tokens and successes refill them. When most calls fail, the bucket empties and the client returns errors immediately instead of retrying. That frees threads and reduces load on the failing service.
Answer: It is not transient. It means the item's state doesn't match your condition, for example the payment is already recorded. Retrying the same write fails the same way. Re-read the item and decide; in an idempotent handler it often means "already done".
Answer: Failures such as missing permissions or a target resource that no longer exists are not retried. They go straight to the target's DLQ if one is configured; otherwise they are dropped. That is why every target should have a DLQ and an alarm.
Interview questions
- "How do you choose retry settings for a service that calls a payment scheme?" — Classify errors first. Retry only transient and throttling errors, and only idempotent operations. Pick one retry owner per hop (usually the queue). Use exponential backoff with full jitter and a cap. Set a timeout ladder so inner timeouts are shorter. Add a circuit breaker and a DLQ with an owner and an alarm. Measure the retry rate.
- "What is a retry storm and how have you prevented one?" — Layered retries multiply load on a failing dependency. Prevent it with a single retry owner, retry budgets (SDK standard mode or a breaker), jitter, load shedding at the edge, and honouring 429/
Retry-After. - "Why is a timeout dangerous for payments?" — It is ambiguous: the operation may have succeeded. Retrying without idempotency can duplicate the payment. Use idempotency keys or query the status before retrying.
- "Standard vs adaptive retry mode in the AWS SDK?" — Standard: backoff with jitter and a retry quota, the right default. Adaptive adds client-side rate limiting that can delay first attempts. It suits a single-resource, throttling-heavy batch client, not a multi-resource service.
Certification practice
A .NET batch job writes millions of records to a single DynamoDB table and receives frequent ProvisionedThroughputExceededException errors. The developer wants the SDK to slow its own request rate automatically when throttling occurs, without writing custom code. What should the developer do?
A. Wrap each PutItem call in a Polly retry with 10 attempts.
B. Set the SDK client's RetryMode to Adaptive for this client.
C. Set MaxErrorRetry to 0 so the job fails fast.
D. Switch to BatchWriteItem and ignore UnprocessedItems.
Answer: B. Adaptive mode adds a client-side rate limiter that reacts to throttling. It fits a client that targets one resource. A multiplies retries on top of the SDK's own and does not lower the request rate. C loses data. D silently drops writes, because unprocessed items must be retried.
A Lambda function consumes an SQS queue. Logs show some messages being processed twice, a few seconds apart, by different invocations. The function's timeout is 60 seconds, typical runtime is 40 seconds, and the queue's visibility timeout is 30 seconds. What is the MOST likely cause?
A. The queue is a standard queue, which always delivers twice. B. The visibility timeout is shorter than the processing time, so messages reappear while still being processed. C. Lambda retries the batch twice by default. D. The DLQ redrives messages automatically.
Answer: B. When processing outlasts the visibility timeout, the message becomes visible and another invocation receives it. AWS recommends at least six times the function timeout. A: standard queues can deliver duplicates, but not always and not this predictably. C describes async invocation, not event source mappings. D: redrive is a manual or explicitly started operation.
A bank's order service calls a partner API that slows down under load. During incidents, retries from many service instances overwhelm the partner. The bank wants to buffer requests durably and control the call rate to the partner. Which design is MOST appropriate?
A. Increase retries in each service instance to 10 with no delay. B. Put requests in an SQS queue and have a consumer with limited concurrency call the partner, using backoff on failure and a DLQ. C. Call the partner synchronously from API Gateway with a longer integration timeout. D. Use SNS to fan out each request to multiple partner endpoints.
Answer: B. The queue buffers work durably. Consumer concurrency caps the call rate. Backoff spaces retries out, and the DLQ captures failures. A causes a retry storm. C keeps the tight coupling and the fragility. D multiplies calls.
Completion checklist
- I can classify an error as transient, throttling, permanent or business, and say whether to retry it
- I can compute a full-jitter backoff delay by hand
- I set
retry_modeandmax_attemptson my profile andRetryModeexplicitly in code - I ran the worker with injected failures and saw growing, randomised delays and a DLQ landing
- I can list each AWS messaging service's default retry behaviour and where failures go
- I deleted the queues and confirmed nothing tagged
Lab=m08-retriesremains
Key takeaways
- Retry only what can succeed next time, and only when repeating it is idempotent.
- Use exponential backoff with full jitter and a cap;
JitterStrategy: FULLin Step Functions is off unless you set it. - Have one retry owner per hop. Layered retries multiply into retry storms.
- The SDK already retries AWS calls. Set
RetryMode = Standardexplicitly and don't wrap SDK calls in more retries. - Queues are the best retry buffer. Back off per message with
ChangeMessageVisibility, cap withmaxReceiveCount, and always attach a DLQ with an alarm and an owner.
Further reading
- Retry behavior (AWS SDKs and Tools Reference Guide)
- Retries and timeouts (AWS SDK for .NET v4 Developer Guide)
- Handling errors in Step Functions workflows
- How EventBridge retries delivering events
- Amazon SNS message delivery retries
- How Lambda handles errors and retries with asynchronous invocation
- Amazon SQS message quotas
- Retry with backoff pattern (AWS Prescriptive Guidance)
- Circuit breaker pattern (AWS Prescriptive Guidance)
- Making retries safe with idempotent APIs (Amazon Builders' Library)
Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.
Amazon EventBridge: buses, rules, schemas and Pipes
Route business events by content with Amazon EventBridge: event buses and rules, patterns and input transformers, target retries and dead-letter queues, archive and replay, the schema registry (and its .NET gap), EventBridge Pipes, and the new enhanced custom event bus, with .NET 10, CDK and Terraform.
Delivery guarantees and ordering
Know exactly what each AWS messaging service promises — at-most-once, at-least-once or exactly-once, ordered or not — and build .NET consumers that stay correct when messages arrive twice, late or out of order, using FIFO message groups, sequence numbers and version checks.