Skip to Content
Cloud Engineering Academy
8. Messaging, Events & Microservices

Dead-letter queues and redrive

Stop one bad message from blocking a payment run: configure SQS dead-letter queues with a sensible maxReceiveCount, use partial batch responses in .NET Lambda handlers, alarm on DLQ depth, and redrive messages safely after a fix.

intermediate⏱ 2 h 30 min🟢 FreeDVA-C03SAA-C03

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS 4.1.0.2) · Amazon.Lambda.SQSEvents 3.0.1 · AWS CLI v2 · CDK v2 (C#) · Terraform AWS provider v6 · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this lesson you can:

  • Explain what a poison message is and why, without a dead-letter queue (DLQ), it either blocks a queue or silently disappears.
  • Configure a redrive policy (maxReceiveCount plus a DLQ ARN) and a redrive allow policy on Amazon SQS, and choose a maxReceiveCount that separates transient failures from permanent ones.
  • Return partial batch responses (ReportBatchItemFailures) from a .NET Lambda function so one bad message does not force nine good ones to be processed again.
  • Raise a CloudWatch alarm the moment anything lands in a DLQ, and explain why DLQ retention must be longer than the source queue's.
  • Redrive messages back to the source queue with StartMessageMoveTask, at a controlled rate, only after the cause is fixed, and explain why redrive is only safe when consumers are idempotent.

Prerequisites

Why this matters in banking

It is 06:10 on the last working day of the month. Harbourline's Payments Service consumes a queue of outbound Faster Payments instructions. Overnight, a partner system started sending one instruction with a creditor IBAN that contains a lowercase letter. The validator throws an exception, the message is not deleted, and after 30 seconds it becomes visible again. The consumer picks it up again. And again.

If the queue is FIFO and that instruction sits at the front of its message group, every later payment from the same debtor account waits behind it. Payroll for a small business customer does not go out. If the queue is standard, the other payments still flow, but the bad message is retried thousands of times until the retention period expires after four days, at which point it vanishes. Nobody is told, and the customer's payment is simply lost.

A dead-letter queue turns both failure modes into a controlled, visible event: after a few attempts the message is moved aside, the rest of the queue keeps flowing, an alarm wakes the on-call engineer, and the message is kept safely until someone fixes the cause and redrives it. In a regulated bank, "we know exactly which instructions failed, and we replayed them after the fix" is the answer the operational-resilience team wants to hear. "We lost one" is not.

The idea in plain language

A queue consumer can fail on a message for two very different reasons:

  1. Transient failure. The database timed out, the downstream API returned 503, the Lambda function ran out of time. Try again in a moment and it will probably work.
  2. Permanent failure. The message is malformed, refers to an account that does not exist, or triggers a bug in your code. Trying again will fail in exactly the same way, forever.

A message that fails permanently is called a poison message. SQS does not know which kind of failure happened. All it sees is that a consumer received the message and did not delete it before the visibility timeout ran out. So SQS counts: every receive increments the message's ApproximateReceiveCount. A redrive policy on the source queue says "once a message has been received more than N times, stop handing it out and move it to this other queue instead." N is maxReceiveCount, and the other queue is the dead-letter queue.

Later, after you fix whatever caused the failure, you redrive: you move the messages from the DLQ back into the source queue so they are processed normally.

Analogy: A bank's post room sorts incoming letters. If a letter can't be delivered (unreadable address), the clerk tries a few times, then puts it in the "returned mail" tray instead of letting it circulate forever and jam the sorting machine. Someone checks the tray every morning, fixes the address, and puts the letter back into the normal flow. The DLQ is the returned-mail tray, maxReceiveCount is "how many tries before it goes in the tray", and redrive is "put it back in the flow".

What problem dead-letter queues solve

Without a DLQWith a DLQ
A poison message is retried until the retention period ends (default 4 days, maximum 14), then deleted silentlyIt is moved aside after maxReceiveCount receives and kept for inspection
On a FIFO queue, a stuck message blocks every later message in the same message groupThe blocking message is removed, so the group moves on (but ordering for that group is now broken — see below)
Each retry costs a Lambda invocation, logs and possibly downstream callsWasted work is bounded to maxReceiveCount attempts
Nobody notices until a customer complainsAn alarm on DLQ depth fires within minutes
No clean way to reprocess after a fixRedrive moves messages back, at a rate you choose

What a DLQ is not. A DLQ is an ordinary SQS queue. There is no special queue type. It becomes a DLQ only because another queue's redrive policy points at it. It must be the same type (a FIFO source needs a FIFO DLQ, a standard source needs a standard DLQ), in the same account and Region as the source queue.

Honest limits and sharp edges (verified 2026-11-18):

FactDetail
Same account and RegionA DLQ cannot be in another account or Region
Same typeFIFO source → FIFO DLQ; standard → standard
maxReceiveCountSet per source queue; AWS warns that very low values (such as 1) send messages to the DLQ after a single failed receive. Allowed range 1 to 1,000 ⚠ VERIFY upper bound
Retention on standard queuesExpiry is based on the original enqueue time. Moving to a DLQ does not reset the clock
Retention on FIFO queuesThe enqueue time is reset when a message moves to the DLQ
Redrive allow policyControls which source queues may use this queue as a DLQ: allowAll (default), byQueue (up to 10 source queue ARNs) or denyAll
Redrive task durationOne redrive task runs for at most 36 hours
Concurrent redrive tasksUp to 100 active tasks per account
Custom redrive velocityUp to 500 messages per second
Redriven messagesGet a new message ID and a new enqueue time; no filtering or editing during redrive
Standard queues with maxReceiveCount greater than 3After 3 failed receives, SQS moves the message to the back of the queue, so ApproximateAgeOfOldestMessage ignores it

The retention row deserves its own example. Suppose the source queue keeps messages for 4 days and the DLQ also keeps them for 4 days. A message spends 3 days failing in the source queue before it is moved. On a standard DLQ it is then deleted after only 1 more day, because the clock started when it was first sent. If the alarm fired on a Friday evening and the team looks on Monday, the evidence is gone. Always set the DLQ retention longer than the source queue's; 14 days (the maximum) is a sensible default.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Service Bus dead-letter subqueue ($DeadLetterQueue)A separate SQS queue referenced by the source queue's redrive policyIn Service Bus every queue has a built-in DLQ; in SQS you create the DLQ yourself and wire it up
MaxDeliveryCount on a Service Bus queue (default 10)maxReceiveCount in the redrive policySame idea. SQS has no DLQ until you configure one, so the default is "retry until retention expires"
DeadLetterReason / DeadLetterErrorDescription propertiesNone set automaticallySQS doesn't record why a message failed. Log the reason yourself, keyed by message ID
Explicit DeadLetterMessageAsync() from the consumerNo direct equivalentYou either let the receive count run out, or send the message to the DLQ yourself and delete the original
Service Bus Explorer "resubmit"DLQ redrive (StartMessageMoveTask, console Start DLQ redrive)AWS's redrive is a managed, rate-limited, asynchronous task
MSMQ "transactional dead-letter queue"SQS DLQSame purpose; no MSDTC transactions in SQS
Azure Functions poison queue (<queue>-poison for Storage queues)SQS DLQ plus Lambda partial batch responsesFunctions creates the poison queue automatically. With Lambda you configure the DLQ on the SQS queue itself

The biggest mental shift for a Service Bus user: SQS does not dead-letter on its own. A fresh queue with no redrive policy will retry a poison message until the retention period ends and then delete it without telling anyone.

Architecture

Security-boundary view. The DLQ holds exactly the same data as the source queue: payment identifiers, amounts and possibly personal data. Treat it with the same controls: the same encryption (SSE-SQS or SSE-KMS with the same key policy), a queue policy that denies non-TLS access, and IAM permissions that let only the operations role receive from it and start redrives. The consumer function needs no access to the DLQ at all: SQS moves messages there itself. Log message IDs and failure reasons, never message bodies.

How a message reaches the DLQ, step by step

It helps to trace one poison message precisely, because exam questions and production incidents both hinge on the details.

  1. The producer sends the message. ApproximateReceiveCount is 0.
  2. Lambda's event source mapping polls the queue and receives a batch including our message. Count is now 1. The message becomes invisible for the visibility timeout (180 seconds in the lab).
  3. The handler fails it (throws, or lists it in batchItemFailures). Lambda does not delete it.
  4. After 180 seconds the message becomes visible. The next poll receives it: count 2.
  5. Steps 3–4 repeat until the count reaches maxReceiveCount (5).
  6. On the next receive attempt, SQS sees that the count has already reached the limit, and instead of delivering the message it moves it to the DLQ. The consumer never sees a sixth delivery.

So a message with maxReceiveCount = 5 is processed five times before it moves, and the move happens lazily when someone next tries to receive. With a 180-second visibility timeout, that is roughly 15 minutes from first failure to DLQ. If you need the alarm faster, you shorten the visibility timeout or lower the count, and both have costs.

Choosing maxReceiveCount. It must be high enough that a short downstream outage does not dump healthy messages into the DLQ, and low enough that poison messages do not waste hours.

SituationSuggested valueReasoning
Lambda consumer (AWS guidance)At least 5Lambda can receive a message and then be throttled before invoking your code; that still counts as a receive
Downstream outages of up to about 10 minutes, 3-minute visibility55 × 3 minutes ≈ 15 minutes of retrying
Long outages expected (e.g. a partner batch window)Higher count or a longer visibility timeout, plus a circuit breakerOtherwise every message ends up in the DLQ and you redrive everything
FIFO queue where ordering matters more than throughputConsider no DLQ, or a DLQ with an operational process to stop the groupMoving a message out of a group lets later messages overtake it

FIFO queues and DLQs: the ordering trade-off

On a FIFO queue, a message that is not deleted blocks every later message with the same message group ID. That is the guarantee you asked for. A DLQ breaks that deadlock by moving the stuck message out, after which message 2, 3 and so on in the group are delivered. For notifications that is exactly what you want. For a ledger it may not be: if message 1 was "open overdraft facility" and message 2 is "debit £500", processing message 2 without message 1 is wrong.

AWS's own guidance is blunt: don't attach a DLQ to a FIFO queue if breaking the exact order of operations is unacceptable. For ledger-like flows at Harbourline, the pattern is:

  • Keep the DLQ (losing a posting is worse than pausing one), and
  • make the consumer check sequence numbers per account, so a later message that arrives without its predecessor is itself failed and also lands in the DLQ, and
  • redrive the whole group in order once fixed.

Delivery guarantees and ordering goes deeper into sequence numbers and out-of-order events.

Two FIFO-specific details:

  • When a message moves to a FIFO DLQ, SQS replaces its deduplication ID with its message ID, so two unrelated failed messages that happened to share a deduplication ID are both kept.
  • Redrive from a FIFO DLQ must go to a FIFO destination. Redriven messages are interleaved with whatever new messages producers are sending at the same time.

Partial batch responses in Lambda

When Lambda reads from SQS through an event source mapping, it receives up to 10 messages per batch by default (up to 10,000 on standard queues with a batching window). Without extra configuration the rule is all-or-nothing:

  • Handler returns normally → Lambda deletes every message in the batch.
  • Handler throws → Lambda deletes none of them. All ten become visible again and all ten have their receive count increased.

That second rule is nasty. One poison message in a batch of ten means nine good messages are processed again (duplicate side effects unless you are idempotent) and all ten move towards the DLQ together. With unlucky batching, healthy messages can end up in the DLQ alongside the poison one.

The fix is partial batch responses:

  1. Turn on ReportBatchItemFailures in the event source mapping's FunctionResponseTypes.
  2. Return an SQSBatchResponse listing the itemIdentifier (the SQS message ID) of each message that failed.
  3. Lambda deletes everything else.

Rules worth memorising:

You returnLambda does
Empty batchItemFailuresDeletes all messages
Some message IDsDeletes the others; failed ones become visible after the timeout
An ID that is not in the batch, or an empty/null IDTreats the whole batch as failed
Function throws an exceptionTreats the whole batch as failed
ReportBatchItemFailures not enabled but you return a responseResponse is ignored; success deletes everything

For FIFO queues, stop processing at the first failure and report that message and every later message in the batch as failed. Otherwise a later message from the same group could be deleted before the earlier failed one is retried, which breaks ordering.

Step-by-step implementation

Create the DLQ first

The DLQ must exist before a redrive policy can point at it. Give it the maximum retention (14 days = 1,209,600 seconds) and the same encryption as the source queue.

Restrict which queues may use it

Set a redrive allow policy of byQueue listing the source queue ARN. That stops an unrelated team from pointing their queue at your DLQ and filling it with their failures (and their personal data).

Attach the redrive policy to the source queue

RedrivePolicy is a JSON string attribute: deadLetterTargetArn plus maxReceiveCount. Choose a visibility timeout of at least six times the Lambda timeout.

Make the consumer report individual failures

Enable ReportBatchItemFailures on the event source mapping and return SQSBatchResponse from the .NET handler. Classify failures: poison (will never succeed) vs transient (may succeed later). Both are reported as failures; the difference is in what you log and whether you also call your downstream again.

Alarm on DLQ depth

Create a CloudWatch alarm on the DLQ's ApproximateNumberOfMessagesVisible, statistic Maximum, threshold greater than 0, one evaluation period of one minute, missing data treated as not breaching. Send it to an SNS topic that pages or emails the on-call team.

Write the runbook before you need it

Who looks at the DLQ, how they inspect messages without leaking personal data, how they decide "fix and redrive" vs "discard", and how they redrive. The module project includes a sample runbook (labs/m08-notification-system/RUNBOOK.md).

Practise a redrive

Inject a poison message, watch it land in the DLQ, deploy a fix (or decide it is bad data), redrive, and confirm no customer got two notifications.

AWS Console instructions

Create the DLQ

  1. Console → Amazon SQS → Queues → Create queue.
  2. Type Standard, name hl-m08-l3-payments-dlq.
  3. Configuration → Message retention period: 14 days.
  4. Encryption: Server-side encryption enabled, Amazon SQS key (SSE-SQS).
  5. Redrive allow policy: Enabled → By queue → add the source queue's ARN (you can come back and add it once the source queue exists).
  6. Add tags Project=cea-lab, Lab=m08-l3, Environment=learn, Owner=<you> → Create queue.

Create the source queue with a redrive policy

  1. Create queue → Standard, name hl-m08-l3-payments.
  2. Visibility timeout: 180 seconds. Retention: 4 days.
  3. Dead-letter queue: Enabled → choose hl-m08-l3-payments-dlq → Maximum receives: 5.
  4. Same encryption and tags → Create queue.

Redrive after a fix

  1. Queues → select hl-m08-l3-payments-dlq → Start DLQ redrive.
  2. Message destination: Redrive to source queue(s).
  3. Velocity control settings: Custom max velocity → 10 messages per second (start low).
  4. Redrive messages. The queue's details page shows progress; Cancel DLQ redrive stops it (messages already moved stay moved).

Alarm

  1. Console → CloudWatch → Alarms → Create alarm → Select metric → SQS → Queue Metrics → hl-m08-l3-payments-dlq / ApproximateNumberOfMessagesVisible.
  2. Statistic Maximum, period 1 minute, condition Greater than 0.
  3. Additional configuration → Missing data treatment → Treat missing data as good (not breaching).
  4. Notification: an SNS topic for the on-call team. Name hl-m08-l3-payments-dlq-not-empty.

AWS CLI

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
TAGS='Project=cea-lab,Lab=m08-l3,Environment=learn,Owner=learner'

# 1. DLQ: 14-day retention (seconds), SSE-SQS encryption
DLQ_URL=$(aws sqs create-queue --queue-name hl-m08-l3-payments-dlq \
  --attributes MessageRetentionPeriod=1209600,SqsManagedSseEnabled=true \
  --tags "$TAGS" --query QueueUrl --output text)
DLQ_ARN=$(aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

# 2. Source queue with a redrive policy (the policy value is a JSON string inside JSON)
cat > source-attrs.json <<EOF
{
  "VisibilityTimeout": "180",
  "MessageRetentionPeriod": "345600",
  "SqsManagedSseEnabled": "true",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"$DLQ_ARN\",\"maxReceiveCount\":\"5\"}"
}
EOF
SRC_URL=$(aws sqs create-queue --queue-name hl-m08-l3-payments \
  --attributes file://source-attrs.json --tags "$TAGS" --query QueueUrl --output text)
SRC_ARN=$(aws sqs get-queue-attributes --queue-url "$SRC_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)

# 3. Only this source queue may use the DLQ
cat > allow.json <<EOF
{ "RedriveAllowPolicy": "{\"redrivePermission\":\"byQueue\",\"sourceQueueArns\":[\"$SRC_ARN\"]}" }
EOF
aws sqs set-queue-attributes --queue-url "$DLQ_URL" --attributes file://allow.json

# 4. Which queues send to this DLQ?
aws sqs list-dead-letter-source-queues --queue-url "$DLQ_URL"

# 5. Simulate a poison message and a consumer that never deletes it.
aws sqs send-message --queue-url "$SRC_URL" --message-body '{"paymentId":"PAY-bad","creditorIban":"gb82west12345698765432"}'
# For a quick drill, receive it 6 times with a 0-second visibility timeout:
for i in 1 2 3 4 5 6; do
  aws sqs receive-message --queue-url "$SRC_URL" --visibility-timeout 0 \
    --message-system-attribute-names ApproximateReceiveCount \
    --query 'Messages[0].Attributes.ApproximateReceiveCount' --output text
done
# The 6th attempt returns None: the message has moved to the DLQ.
aws sqs get-queue-attributes --queue-url "$DLQ_URL" --attribute-names ApproximateNumberOfMessages

# 6. Redrive back to the source queue at 10 messages per second
aws sqs start-message-move-task --source-arn "$DLQ_ARN" --max-number-of-messages-per-second 10
aws sqs list-message-move-tasks --source-arn "$DLQ_ARN" --max-results 1
# Cancel a running task if you redrove too early:
# aws sqs cancel-message-move-task --task-handle <TaskHandle from start-message-move-task>

# 7. Alarm on DLQ depth (add --alarm-actions <sns-topic-arn> to be notified)
aws cloudwatch put-metric-alarm --alarm-name hl-m08-l3-payments-dlq-not-empty \
  --namespace AWS/SQS --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=hl-m08-l3-payments-dlq \
  --statistic Maximum --period 60 --evaluation-periods 1 \
  --threshold 0 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching

Notes on the commands:

  • RedrivePolicy and RedriveAllowPolicy are strings containing JSON, which is why the inner quotes are escaped. Writing them into a file avoids PowerShell and Bash quoting pain.
  • In step 5 a visibility timeout of 0 makes the message visible again immediately. Never use 0 in a real consumer: it means "anybody can grab this message right now while I am still processing it".
  • start-message-move-task without --destination-arn sends each message back to the queue it came from. That only works for messages that arrived in the DLQ through a redrive policy; messages you put there by hand with send-message have no recorded source and need an explicit destination.

Infrastructure as Code

PaymentsQueueStack.cs
using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.CloudWatch.Actions;
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.SQS;
using Constructs;

namespace Harbourline.Payments.Infra;

public sealed class PaymentsQueueStack : Stack
{
    public PaymentsQueueStack(Construct scope, string id, IStackProps? props = null) : base(scope, id, props)
    {
        var dlq = new Queue(this, "PaymentsDlq", new QueueProps
        {
            QueueName = "hl-m08-l3-payments-dlq",
            RetentionPeriod = Duration.Days(14),           // longer than the source queue
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
        });

        var queue = new Queue(this, "PaymentsQueue", new QueueProps
        {
            QueueName = "hl-m08-l3-payments",
            VisibilityTimeout = Duration.Seconds(180),    // 6 x a 30-second Lambda timeout
            RetentionPeriod = Duration.Days(4),
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 },
        });

        // Redrive allow policy: only our source queue may dead-letter into this DLQ.
        var cfnDlq = (CfnQueue)dlq.Node.DefaultChild!;
        cfnDlq.RedriveAllowPolicy = new Dictionary<string, object>
        {
            ["redrivePermission"] = "byQueue",
            ["sourceQueueArns"] = new[] { queue.QueueArn },
        };

        var onCall = new Topic(this, "OnCall", new TopicProps { TopicName = "hl-m08-l3-oncall", EnforceSSL = true });

        var alarm = new Alarm(this, "DlqNotEmpty", new AlarmProps
        {
            AlarmName = "hl-m08-l3-payments-dlq-not-empty",
            AlarmDescription = "Payments failed after 5 attempts. See the runbook before redriving.",
            Metric = dlq.MetricApproximateNumberOfMessagesVisible(new MetricOptions
            {
                Period = Duration.Minutes(1),
                Statistic = "Maximum",
            }),
            Threshold = 0,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            EvaluationPeriods = 1,
            TreatMissingData = TreatMissingData.NOT_BREACHING,
        });
        alarm.AddAlarmAction(new SnsAction(onCall));
    }
}

Wait: there is a circular reference in this snippet. The DLQ's allow policy refers to the source queue's ARN, and the source queue's redrive policy refers to the DLQ's ARN. CloudFormation cannot create two resources that each need the other's ARN. Two ways out:

  • Build the source queue ARN as a string from its name instead of a token: $"arn:aws:sqs:{Region}:{Account}:hl-m08-l3-payments" (it works because you chose the name), or
  • Leave the allow policy at its default (allowAll) and restrict access with IAM instead.
["sourceQueueArns"] = new[] { $"arn:{Partition}:sqs:{Region}:{Account}:hl-m08-l3-payments" },

Spotting dependency cycles like this is part of reading IaC critically. cdk synth would report it as Circular dependency between resources.

Complete code

Two pieces of runnable .NET code: the Lambda handler that reports partial batch failures, and an operations console tool that inspects a DLQ and starts a redrive. The full, tested version of the handler is in labs/m08-notification-system/src/Harbourline.Notifications.Functions.

The handler: classify, report, never throw for one message

PaymentInstructionFunction.cs
using System.Text.Json;
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Harbourline.Payments.Functions;

public sealed record PaymentInstruction(string PaymentId, string DebtorAccountId, string CreditorIban, decimal Amount, string Currency);

/// <summary>Thrown for failures that will never succeed on retry.</summary>
public sealed class PoisonMessageException(string reason) : Exception(reason);

public sealed class PaymentInstructionFunction
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);

    public async Task<SQSBatchResponse> HandleAsync(SQSEvent sqsEvent, ILambdaContext context)
    {
        var failures = new List<SQSBatchResponse.BatchItemFailure>();
        var isFifo = sqsEvent.Records is { Count: > 0 } records
                     && records[0].EventSourceArn.EndsWith(".fifo", StringComparison.Ordinal);

        foreach (var record in sqsEvent.Records ?? [])
        {
            // FIFO: once one message fails, fail the rest of the batch to keep group order.
            if (isFifo && failures.Count > 0)
            {
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
                continue;
            }

            try
            {
                var instruction = Parse(record.Body);
                await SubmitAsync(instruction);
            }
            catch (PoisonMessageException ex)
            {
                Log(context, record, "poison", ex.Message);
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
            }
            catch (Exception ex)
            {
                Log(context, record, "transient", ex.GetType().Name);
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
            }
        }

        return new SQSBatchResponse { BatchItemFailures = failures };
    }

    private static PaymentInstruction Parse(string body)
    {
        PaymentInstruction? instruction;
        try
        {
            instruction = JsonSerializer.Deserialize<PaymentInstruction>(body, Json);
        }
        catch (JsonException)
        {
            throw new PoisonMessageException("body is not valid JSON");
        }

        if (instruction is null || !instruction.PaymentId.StartsWith("PAY-", StringComparison.Ordinal))
            throw new PoisonMessageException("paymentId missing or malformed");
        if (instruction.CreditorIban != instruction.CreditorIban.ToUpperInvariant())
            throw new PoisonMessageException("creditor IBAN is not upper case");
        if (instruction.Amount <= 0m)
            throw new PoisonMessageException("amount must be positive");
        return instruction;
    }

    // Stand-in for the call to the payment scheme gateway. Must be idempotent on PaymentId.
    private static Task SubmitAsync(PaymentInstruction instruction) => Task.CompletedTask;

    private static void Log(ILambdaContext context, SQSEvent.SQSMessage record, string kind, string reason)
    {
        var receiveCount = record.Attributes?.GetValueOrDefault("ApproximateReceiveCount") ?? "?";
        // Identifiers and reasons only. Never log the body: it contains an IBAN.
        context.Logger.LogWarning(JsonSerializer.Serialize(new { messageId = record.MessageId, receiveCount, kind, reason }));
    }
}

The operations tool: peek, summarise, redrive

DlqTool/Program.cs
using Amazon.SQS;
using Amazon.SQS.Model;

// Usage (AWS_PROFILE=learn-sandbox, AWS_REGION=us-east-1):
//   dotnet run -- inspect hl-m08-l3-payments-dlq
//   dotnet run -- redrive hl-m08-l3-payments-dlq 10
//   dotnet run -- status  hl-m08-l3-payments-dlq
if (args.Length < 2)
{
    Console.WriteLine("Usage: inspect|redrive|status <dlq-name> [messagesPerSecond]");
    return 1;
}

using var sqs = new AmazonSQSClient();
var dlqUrl = (await sqs.GetQueueUrlAsync(new GetQueueUrlRequest { QueueName = args[1] })).QueueUrl;
var attrs = await sqs.GetQueueAttributesAsync(new GetQueueAttributesRequest
{
    QueueUrl = dlqUrl,
    AttributeNames = ["QueueArn", "ApproximateNumberOfMessages"],
});
var dlqArn = attrs.Attributes["QueueArn"];

switch (args[0])
{
    case "inspect":
        // Receive with a short visibility timeout: we look, we do not delete.
        var response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
        {
            QueueUrl = dlqUrl,
            MaxNumberOfMessages = 10,
            VisibilityTimeout = 30,
            WaitTimeSeconds = 2,
            MessageSystemAttributeNames = ["ApproximateReceiveCount", "SentTimestamp", "DeadLetterQueueSourceArn"],
        });

        Console.WriteLine($"Approximate depth: {attrs.Attributes["ApproximateNumberOfMessages"]}");
        foreach (var m in response.Messages ?? [])
        {
            var sent = m.Attributes is not null && m.Attributes.TryGetValue("SentTimestamp", out var ts)
                ? DateTimeOffset.FromUnixTimeMilliseconds(long.Parse(ts, System.Globalization.CultureInfo.InvariantCulture)).ToString("u")
                : "?";
            var source = m.Attributes?.GetValueOrDefault("DeadLetterQueueSourceArn") ?? "(sent directly)";
            // Show size and a hash, not the content: the body may hold personal data.
            var hash = Convert.ToHexString(System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(m.Body)))[..12];
            Console.WriteLine($"{m.MessageId}  sent {sent}  bytes {m.Body.Length,6}  sha256 {hash}  from {source}");
        }
        return 0;

    case "redrive":
        var rate = args.Length > 2 ? int.Parse(args[2], System.Globalization.CultureInfo.InvariantCulture) : 10;
        var task = await sqs.StartMessageMoveTaskAsync(new StartMessageMoveTaskRequest
        {
            SourceArn = dlqArn,                      // the DLQ is the SOURCE of a move task
            MaxNumberOfMessagesPerSecond = rate,     // omit for "system optimised"
            // DestinationArn omitted: each message goes back to the queue it came from.
        });
        Console.WriteLine($"Started move task {task.TaskHandle} at {rate} msg/s");
        return 0;

    case "status":
        var tasks = await sqs.ListMessageMoveTasksAsync(new ListMessageMoveTasksRequest { SourceArn = dlqArn, MaxResults = 3 });
        foreach (var t in tasks.Results ?? [])
        {
            Console.WriteLine($"{t.Status,-10} moved {t.ApproximateNumberOfMessagesMoved}/{t.ApproximateNumberOfMessagesToMove} {t.FailureReason}");
        }
        return 0;

    default:
        Console.WriteLine($"Unknown command {args[0]}");
        return 1;
}
DlqTool/DlqTool.csproj
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="AWSSDK.SQS" Version="4.1.0.2" />
  </ItemGroup>
</Project>

Code explained line by line

Handler

  1. [assembly: LambdaSerializer(...)] — tells the Lambda runtime how to turn the incoming JSON event into SQSEvent and the returned SQSBatchResponse back into JSON.
  2. HandleAsync returns Task<SQSBatchResponse> — this return type is what makes partial batch responses possible. It only has an effect when ReportBatchItemFailures is enabled on the event source mapping.
  3. isFifo — the event source ARN of a FIFO queue ends in .fifo. For FIFO we stop after the first failure and fail everything after it, so a later message in the same group can't be deleted ahead of an earlier failed one.
  4. The try wraps one message. An exception for one message never escapes the loop; escaping would fail the whole batch.
  5. PoisonMessageException vs Exception — both are reported as failures. The difference is in the log: on-call engineers can count poison vs transient failures with a Logs Insights query and immediately see whether they face bad data or an outage.
  6. Parse — validation rules that will fail identically on every retry are poison by definition. A lowercase IBAN is our month-end scenario.
  7. SubmitAsync — the comment matters: because SQS redelivers, and because redrive replays, the submission must be idempotent on PaymentId. See Idempotency in .NET.
  8. Log — writes structured JSON with the message ID and receive count. It deliberately leaves out the body, which contains an IBAN.

Operations tool

  1. GetQueueUrlAsync / GetQueueAttributesAsync — turns the name into a URL and fetches the ARN (needed by the move-task APIs) and depth. In SDK v4, AttributeNames is a List<string>, and collection expressions ([...]) keep it short.
  2. ReceiveMessageAsync with VisibilityTimeout = 30 — receiving from a DLQ hides messages for that period. Keep it short so an operator's peek doesn't interfere with a redrive that starts a minute later.
  3. MessageSystemAttributeNames — the SDK v4 property for system attributes. DeadLetterQueueSourceArn tells you which source queue the message came from.
  4. response.Messages ?? [] — SDK v4 returns null for empty collections, not an empty list. Forgetting this is the most common v3-to-v4 migration bug.
  5. The SHA-256 prefix lets two engineers confirm they are looking at the same message without pasting its content into a chat channel.
  6. StartMessageMoveTaskAsync — note that the DLQ is the source of a move task. The rate is capped by MaxNumberOfMessagesPerSecond; omitting it lets SQS choose ("system optimised").
  7. ListMessageMoveTasksAsync — returns up to the 10 most recent tasks for the queue with status (RUNNING, COMPLETED, CANCELLING, CANCELLED, FAILED) and counts.

Security considerations

⚠ Learning shortcut — not for production

The CLI drill uses your sandbox administrator role to receive from the DLQ and to start a redrive. In production, create a dedicated operations role with only sqs:ReceiveMessage, sqs:GetQueueAttributes, sqs:StartMessageMoveTask, sqs:CancelMessageMoveTask, sqs:ListMessageMoveTasks, sqs:ListDeadLetterSourceQueues and sqs:DeleteMessage on the DLQ ARN, plus sqs:SendMessage on the destination queue (and kms:Decrypt / kms:GenerateDataKey if you use SSE-KMS). Require MFA through IAM Identity Center and log every redrive in CloudTrail.

  • Same protection as the source. A DLQ full of failed payment instructions is a concentrated store of personal and financial data. Encrypt it with the same key, restrict who can read it, and keep its retention as short as your process allows (14 days is the maximum, not a requirement).
  • Redrive allow policy. Use byQueue so that only your own source queues can dead-letter into your DLQ.
  • Don't log bodies. Log message IDs, receive counts, error categories and hashes. If engineers need to see content, do it in a controlled tool with access logging, not by copying JSON into tickets.
  • Redrive is a privileged action. It can trigger real side effects (payments, notifications). Treat it like a change: a ticket, a reason, and a second pair of eyes for large redrives.
  • Discarding is also a decision. Purging a DLQ deletes evidence. Agree with the data owner and record the decision; for payment instructions, the business usually needs to tell the customer.
  • GDPR / UK GDPR. Retention of personal data in DLQs and archives needs review by your data-protection team. This lesson does not make a design compliant.

Banking use case

Harbourline's Notifications Service (the module project) runs one SQS queue per channel. In the first week of production:

  • Monday: a publisher bug sends currency: "POUNDS" for 37 transactions. The handler classifies them as poison. After five receives each they land in hl-m08-notify-sms-dlq, the alarm fires at 09:14, and the on-call engineer's Logs Insights query shows 37 × poison: currency is not a 3-letter ISO 4217 code. The publishing team fixes their mapping. The 37 messages themselves are still invalid, so they are not redriven. Instead the publisher re-sends corrected events.
  • Wednesday: the SMS provider has a 40-minute outage. Each message is retried five times over 15 minutes, then 2,300 messages land in the DLQ. When the provider recovers, the engineer redrives at 20 messages per second to avoid flooding it. The idempotency table ensures that the 12 messages that were actually delivered just before the provider's API started timing out are not sent twice.
  • Lesson learned: for the outage case, maxReceiveCount = 5 with a 3-minute visibility timeout was too eager. The team adds a circuit breaker: when the provider is failing, the handler stops calling it and reports all messages as failed quickly. They also consider a longer visibility timeout for that queue. Retry strategies covers the design.

Common mistakes

  • No DLQ at all. Poison messages loop until retention expires and then vanish. → Every production queue gets a DLQ and an alarm; make it a cdk-nag or policy-as-code rule (cdk-nag's AwsSolutions-SQS3 flags queues without one).
  • DLQ retention equal to or shorter than the source. On standard queues messages expire based on the original enqueue time and disappear from the DLQ early. → 14 days on DLQs.
  • maxReceiveCount = 1. One Lambda throttle or a single timeout sends a healthy message to the DLQ. → At least 5 for Lambda consumers.
  • Throwing from the Lambda handler for one bad message. The whole batch is retried; good messages are reprocessed and pushed towards the DLQ. → Partial batch responses.
  • Returning batchItemFailures without enabling ReportBatchItemFailures. Lambda ignores the response and deletes every message on success. The bad message is silently lost. → Enable it on the event source mapping and test it.
  • Redriving before fixing. Messages fail five more times and come straight back, and you have spent another 15 minutes. → Redrive only after a fix, or after confirming the outage is over.
  • Redriving into a non-idempotent consumer. Some messages in the DLQ were partly processed (the side effect happened, then the delete failed). Redrive repeats the side effect. → Idempotency keys based on business identifiers.
  • Alarming on the source queue's age only. With maxReceiveCount above 3 on standard queues, SQS moves repeatedly failing messages to the back, and ApproximateAgeOfOldestMessage stops reflecting them. → Alarm on DLQ depth as well.
  • One DLQ shared by many unrelated queues. You can't tell where failures came from or redrive selectively. → One DLQ per source queue (the DeadLetterQueueSourceArn attribute helps, but separate queues are clearer).

Troubleshooting

SymptomLikely causeFix
Poison message is retried forever, DLQ stays emptyNo redrive policy on the source queue, or it points at the wrong ARNaws sqs get-queue-attributes --attribute-names RedrivePolicy; check the ARN and count
InvalidParameterValue when setting RedrivePolicyDLQ is a different type (FIFO vs standard) or in another Region/accountCreate a matching DLQ in the same account and Region
Setting the redrive policy fails with a permission-style errorDLQ's redrive allow policy is denyAll or byQueue without this sourceAdd the source ARN to sourceQueueArns
Messages vanish from the DLQ sooner than its retentionStandard queue: expiry counts from the original enqueue timeIncrease DLQ retention to 14 days
Good messages show up in the DLQWhole batch failing (no partial batch responses), visibility timeout shorter than processing time, or maxReceiveCount too lowEnable ReportBatchItemFailures; visibility ≥ 6 × function timeout; count ≥ 5
Redrive task FAILED with an access errorMissing sqs:SendMessage on destination or kms:Decrypt/kms:GenerateDataKey for encrypted queuesAdd the permissions to the operator role / key policy
start-message-move-task without a destination failsMessages were sent to the DLQ manually, so there is no recorded source queuePass --destination-arn explicitly
DLQ alarm never fires even though messages arriveWrong metric (e.g. NumberOfMessagesSent is not emitted for messages moved by redrive policy), or missing data treated as breaching/ignored inconsistentlyUse ApproximateNumberOfMessagesVisible with Maximum ⚠ VERIFY metric behaviour for moved messages
Messages redriven but processed twiceConsumer is not idempotentAdd an idempotency table keyed on a business ID

Cost and cleanup

Cost: free at lab volumes. A DLQ is an ordinary SQS queue: you pay only for requests, and the first 1 million SQS requests per month are free. Moving a message to a DLQ and redriving it each count as requests. The CloudWatch alarm is one standard alarm; the CloudWatch free tier includes 10 alarms per month, and a standard alarm costs about USD 0.10 per month beyond that ⚠ VERIFY on the CloudWatch pricing page. This lesson's drill uses a few dozen requests.

Free alternative: the module project's offline simulator (dotnet run --project src/Harbourline.Notifications.Cli -- simulate) reproduces receive counts, DLQ moves and redrive without an AWS account.

Teardown:

export AWS_PROFILE=learn-sandbox AWS_REGION=us-east-1
aws cloudwatch delete-alarms --alarm-names hl-m08-l3-payments-dlq-not-empty
for q in hl-m08-l3-payments hl-m08-l3-payments-dlq; do
  url=$(aws sqs get-queue-url --queue-name "$q" --query QueueUrl --output text) && aws sqs delete-queue --queue-url "$url"
done
rm -f source-attrs.json allow.json

# Nothing tagged for this lesson should remain (tag data may lag a few minutes)
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project,Values=cea-lab Key=Lab,Values=m08-l3 \
  --query 'ResourceTagMappingList[].ResourceARN'

With CDK: cdk destroy. With Terraform: terraform destroy.

Exercises

  • 🟢 Guided: Run the CLI section end to end. Record the ApproximateReceiveCount printed by each of the six receives, confirm the message is in the DLQ, then redrive it and confirm it is back in the source queue with a new message ID.
  • 🟡 Challenge: Deploy the handler above with an SQS event source mapping (batch size 10, ReportBatchItemFailures on). Send 9 valid instructions and 1 with a lowercase IBAN in one go. Prove from CloudWatch Logs that the 9 were processed exactly once and only the bad one was retried.
  • 🔴 Live problem: Ticket from the payments operations lead: "Since last night's release the payments DLQ alarm has fired three times. Each time someone redrove the DLQ and the alarm fired again 15 minutes later. 4,100 instructions are now in the DLQ, including some that look perfectly valid. The cut-off for today's Faster Payments submission window is 15:00. What do we do?" Write your incident plan.

Assignment

Add a DLQ, redrive allow policy and depth alarm to a queue in your own portfolio project (or to the hl-m08-l3-payments queue). Include in the README: your chosen maxReceiveCount and visibility timeout with the reasoning, a one-page runbook (when to redrive, when to discard, who approves), and a screenshot-free description of a drill you ran with timestamps.

Solutions

Knowledge check

Interview questions

  • "How do you handle poison messages in SQS?" — A redrive policy with a DLQ and a maxReceiveCount sized to the retry budget (at least 5 with Lambda), partial batch responses so one bad message doesn't fail a batch, an alarm on DLQ depth, a runbook, and redrive after the fix. Consumers are idempotent so redrive is safe.
  • "What's the difference between a transient and a permanent failure, and does SQS know?" — Transient failures may succeed later (timeouts, 503s); permanent ones never will (bad data, bugs). SQS can't tell them apart: it only counts receives. Your code should classify them for logging and alerting, and may short-circuit permanent ones, but both end up as "not deleted".
  • "A message was redriven and processed twice. Why, and how do you prevent it?" — The message may have been partly processed before failing (side effect done, delete not), and redriven messages get a new message ID. Prevent it with an idempotency key derived from a business identifier, stored with a conditional write.
  • "Why might DLQ messages disappear early?" — Standard queue expiry uses the original enqueue time. If the DLQ's retention isn't longer than the source's, messages can expire soon after arriving.
  • "How would you redrive 50,000 messages after an outage?" — Confirm the downstream is healthy, test one message, then redrive with a capped velocity, watch error rates and the DLQ, and be ready to cancel the move task. Make sure the consumer's concurrency is capped too (event source mapping maximum concurrency).

Certification practice

Completion checklist

  • I can explain poison messages, maxReceiveCount and redrive to a colleague in two minutes
  • I created a queue with a DLQ, redrive allow policy and depth alarm
  • I watched a message reach the DLQ after exactly maxReceiveCount receives
  • I redrove a message and saw that it came back with a new message ID
  • I can explain partial batch responses and the FIFO rule for them
  • I ran the teardown and confirmed nothing tagged Lab=m08-l3 remains

Key takeaways

  • SQS never dead-letters by itself. Every production queue needs a redrive policy, a DLQ of the same type in the same account and Region, and an alarm on DLQ depth.
  • Choose maxReceiveCount as a retry budget (at least 5 with Lambda) and set DLQ retention longer than the source: standard-queue expiry counts from the original send time.
  • Use partial batch responses in Lambda so one poison message never re-runs nine good ones; on FIFO, fail everything after the first failure.
  • Redrive only after a fix, at a controlled rate, into idempotent consumers. Redriven messages get new message IDs.
  • A DLQ is a store of sensitive data: same encryption, least-privilege access, no bodies in logs, and a documented decision whenever you discard.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page