Skip to Content
Cloud Engineering Academy
17. Daily-Work Playbooks & Career

Playbook: SQS messages stuck or redelivered

Diagnose SQS messages that never finish, keep reappearing or end up in a dead-letter queue by reading visibility timeout, receive count, consumer errors and FIFO group blocking, then inspect and redrive safely with idempotent handlers.

intermediate⏱ 1 h 15 min🟢 FreeDVA-C03SOA-C03SAA-C03DOP-C02

Assumes: .NET 10 LTS · AWS SDK for .NET v4 (AWSSDK.SQS) · AWS CLI v2 · Amazon SQS standard and FIFO · Region us-east-1. Verified against official sources on 2026-11-18.

Learning objectives

By the end of this playbook you can:

  • Explain, step by step, what happens to a message from SendMessage to DeleteMessage, including what the visibility timeout and receive count do.
  • Tell apart the four symptoms: messages not arriving, stuck invisible, redelivered repeatedly and landing in the dead-letter queue (DLQ).
  • Read the queue's CloudWatch metrics and attributes to identify the symptom in under five minutes.
  • Fix the classic causes: a visibility timeout shorter than the work, a Lambda concurrency or batch mismatch, a poison message, and a blocked FIFO message group.
  • Inspect a DLQ without destroying evidence and redrive messages at a controlled rate.
  • Run a simulation that reproduces duplicate processing and the three fixes, with no AWS account.

Prerequisites

Why this matters in banking

A queue named hl-payment-instructions feeds the worker that submits payments to the clearing partner. At 15:20 the operations dashboard shows the queue growing and a handful of customers report that a payment was taken twice. Two opposite-looking symptoms, one root cause: the worker's partner call sometimes takes 45 seconds, the queue's visibility timeout is 30 seconds, so the message reappears while the first worker is still busy and a second worker picks it up. Meanwhile one malformed instruction fails every time, is retried until it lands in the DLQ, and nobody is watching the DLQ.

SQS standard queues promise at least once delivery. Duplicates are not a bug in SQS, they are the contract. A bank-grade consumer must therefore be idempotent (safe to run twice) and must treat the queue's retry settings as part of the design, not as defaults.

The idea in plain language

SQS does not hand a message to a consumer and forget it. It lends the message:

  1. A consumer calls ReceiveMessage. SQS makes the message invisible to other consumers for the visibility timeout and increments its receive count.
  2. If the consumer finishes, it calls DeleteMessage and the message is gone.
  3. If the consumer fails, crashes or is too slow, the timeout expires and the message becomes visible again and is delivered to somebody (maybe the same worker).
  4. If a message has been received more times than the queue's maxReceiveCount, SQS moves it to the dead-letter queue instead.

Every strange SQS symptom is one of these four moves going differently from what you expected.

Analogy: A bank's paper work-tray. A clerk takes a form and puts a "being worked on" sign on the tray slot for 30 minutes. If the clerk finishes, the form is filed (deleted). If the clerk goes to lunch, after 30 minutes the sign expires and another clerk takes it. If the same form has gone round five clerks and none could process it, the supervisor moves it to the exceptions tray (the DLQ). Two clerks may both process a form when the first one was only slow, not gone.

Delivery vocabulary

TermMeaning
VisibleAvailable to be received. Counted by ApproximateNumberOfMessagesVisible
In flight / not visibleReceived but not yet deleted, within its visibility timeout. Counted by ApproximateNumberOfMessagesNotVisible
Receive countHow many times the message has been received. Available as the ApproximateReceiveCount attribute
maxReceiveCountThe redrive policy threshold, after which the message moves to the DLQ
Retention periodHow long an unprocessed message is kept (default 4 days, up to 14)
DelaySeconds a new message stays invisible before first delivery
Message group (FIFO)Messages with the same group ID are delivered one at a time, in order

What problem the AWS service solves

SQS decouples the producer from the consumer so a slow or failing consumer does not break callers, and it absorbs bursts. The cost of that convenience is that you own consumer behaviour: how long processing may take, what happens on failure, what to do with poison messages and how to avoid duplicate effects.

Limits and sharp edges (⚠ VERIFY current values before quoting): the visibility timeout defaults to 30 seconds with a maximum of 12 hours; long polling waits up to 20 seconds; the number of in-flight messages is capped (for standard queues in the order of 120,000; for FIFO queues 20,000); a FIFO queue deduplicates by deduplication ID within a five-minute window; a DLQ for a standard queue must be a standard queue and for FIFO must be FIFO; and messages moved to a DLQ keep their original enqueue time, so the DLQ's retention period counts from then.

Microsoft / Azure comparison

You know (Microsoft/Azure)AWS equivalentWhat's different
Service Bus lock duration (peek-lock)SQS visibility timeoutSimilar, but SQS has no explicit "abandon" or lock renewal: you call ChangeMessageVisibility to extend
Service Bus MaxDeliveryCount → dead-letter sub-queueSQS maxReceiveCount → separate DLQ queueThe DLQ is an ordinary queue you create and monitor
Service Bus sessionsSQS FIFO message groupsA failing message blocks its group, not the whole queue
Service Bus duplicate detectionFIFO deduplication ID (5 minutes)Standard queues have no dedup at all
Azure Storage Queue DequeueCountApproximateReceiveCountSame idea. The word "approximate" is literal
Azure Functions queue trigger poison queueLambda event source mapping with a DLQ or an on-failure destinationYou configure the redrive and partial batch failure yourself
Resubmit from the Service Bus dead-letter in the portalSQS DLQ redrive (console and API)Redrive can move messages back at a limited rate

Architecture

The life of one message

Symptom to cause map

Step-by-step implementation

Minutes 0 to 5: identify the symptom from numbers

Read these queue attributes and metrics together:

Metric or attributeWhat it tells you
ApproximateNumberOfMessagesVisibleWaiting to be picked up. Rising means consumers are too few, too slow or stopped
ApproximateNumberOfMessagesNotVisibleIn flight. High and stuck means consumers are not deleting
ApproximateAgeOfOldestMessageHow long the oldest message has waited. This is your user-facing delay
NumberOfMessagesReceived versus NumberOfMessagesDeletedReceived much greater than deleted means redelivery or failures
DLQ ApproximateNumberOfMessagesVisibleAnything above 0 is an incident for a payments queue
Queue attributes VisibilityTimeout, RedrivePolicy, ReceiveMessageWaitTimeSecondsThe configuration you are diagnosing

Compare received to deleted. If they are close, the queue is healthy but behind. If received greatly exceeds deleted, messages are coming back.

Minutes 5 to 20: check the consumer side

  1. Are consumers running and polling? Check the ECS service desired and running counts, or the Lambda event source mapping state.
  2. Do they have sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility and sqs:GetQueueAttributes on the queue, and kms:Decrypt if the queue uses a customer managed key? A missing KMS permission makes every receive fail.
  3. Look at the consumer logs for the message ID and its receive count. Repeating errors with the same ID mean a poison message. Many different IDs, one delivery each but long processing times, mean the visibility timeout is the issue.
  4. For Lambda consumers: check throttles, errors and the function timeout. AWS guidance is to set the queue visibility timeout to at least six times the function timeout (⚠ VERIFY current guidance), and to keep maxReceiveCount comfortably above one so brief throttling does not push messages to the DLQ.

Minutes 20 to 40: pick the cause

ObservationCauseFix
Processing time is longer than the visibility timeoutMessage reappears while still processedRaise the timeout above worst-case processing, or extend it with a heartbeat while working
Same message ID with rising receive count, handler throwsPoison messageFix the code or data, move aside, never loop forever
Lambda receive counts jump during traffic spikes without handler errorsThrottling returns messages to the queueSet reserved or maximum concurrency sensibly, raise maxReceiveCount
Batch of 10: one failing record makes all 10 retryWhole batch fails on one errorEnable partial batch responses (ReportBatchItemFailures)
NotVisible high, nothing completingConsumers fetch but never delete (code path forgets DeleteMessage)Delete on success in every path, add a test
FIFO queue stalled, one group stuckA failing message blocks its group while others flowFix or move the message, use more groups to limit blast radius
Messages vanish after four daysRetention period expiredIncrease retention, consume faster, DLQ retention longer than source
Consumers poll but receive nothingLong polling not used with many empty receives, or wrong queue URL, or a delay on the queueCheck ReceiveMessageWaitTimeSeconds, queue URL and delay

Minutes 40 to 60: inspect the DLQ safely, then fix

  1. Look, do not drain. Receive a few messages from the DLQ for inspection with a long visibility timeout so nobody else sees them, record the message ID, body shape (not full personal data in chat), attributes and ApproximateReceiveCount, and let them return. Receiving does not delete.
  2. Find the failure reason in the consumer logs by message ID.
  3. Fix the cause first. Redriving unchanged messages into a consumer that still fails just fills the DLQ again.

Minutes 60 to 75: redrive deliberately and prevent

Redrive from the DLQ back to the source queue (console Start DLQ redrive, or the start-message-move-task command) at a controlled rate, with the idempotent consumer ready so any message that was already processed has no second effect. Then add the preventive controls: an alarm on DLQ depth above zero, an alarm on oldest message age, a visibility timeout derived from measured processing time, partial batch responses, and an idempotency guard in every handler. Record the message IDs redriven and the outcome for audit.

AWS Console instructions

  1. Console → SQS → Queues → (queue) → Monitoring: visible, not visible, oldest message age, received and deleted.
  2. Console → SQS → (queue) → Details: visibility timeout, retention, delivery delay, receive message wait time, encryption and the Dead-letter queue section.
  3. Console → SQS → (queue) → Send and receive messages → Poll for messages: view message bodies and attributes (including Receive count) without deleting them. Keep the poll duration short.
  4. Console → SQS → (DLQ) → Start DLQ redrive: choose the destination and a velocity setting (system optimised or custom messages per second), then watch the redrive status.
  5. Console → Lambda → (function) → Configuration → Triggers → SQS trigger: batch size, batch window, maximum concurrency and Report batch item failures.

AWS CLI

# 1. Queue configuration and current depth in one call.
aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions \
  --attribute-names VisibilityTimeout MessageRetentionPeriod RedrivePolicy ReceiveMessageWaitTimeSeconds \
    ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessagesDelayed \
  --profile learn-sandbox

# 2. Age of the oldest message (seconds), last hour.
aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=hl-payment-instructions \
  --start-time $(date -u -d '1 hour ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Maximum --profile learn-sandbox

# 3. Peek at DLQ messages WITHOUT deleting: long visibility timeout so nobody else grabs them.
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions-dlq \
  --max-number-of-messages 3 --visibility-timeout 300 --wait-time-seconds 5 \
  --attribute-names All --message-attribute-names All --profile learn-sandbox

# 4. Extend visibility of a message you are inspecting (the receipt handle comes from step 3).
aws sqs change-message-visibility \
  --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions-dlq \
  --receipt-handle "<receipt-handle>" --visibility-timeout 900 --profile learn-sandbox

# 5. Redrive from the DLQ back to the source queue at a limited rate.
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:111122223333:hl-payment-instructions-dlq \
  --destination-arn arn:aws:sqs:us-east-1:111122223333:hl-payment-instructions \
  --max-number-of-messages-per-second 5 --profile learn-sandbox

# 6. Watch the redrive task, and cancel it if the consumer starts failing again.
aws sqs list-message-move-tasks \
  --source-arn arn:aws:sqs:us-east-1:111122223333:hl-payment-instructions-dlq --profile learn-sandbox
# aws sqs cancel-message-move-task --task-handle "<task-handle>" --profile learn-sandbox

Never use purge-queue on a payments queue during an investigation: it deletes every message, cannot be undone and destroys evidence.

Infrastructure as Code

The settings that prevent this incident: a visibility timeout based on the work, a DLQ with longer retention, a sensible maxReceiveCount, encryption, and alarms that watch the DLQ and the age of the oldest message.

using Amazon.CDK;
using Amazon.CDK.AWS.CloudWatch;
using Amazon.CDK.AWS.SQS;
using Constructs;

public class PaymentQueueStack : Stack
{
    public PaymentQueueStack(Construct scope, string id, IStackProps props) : base(scope, id, props)
    {
        var dlq = new Queue(this, "InstructionsDlq", new QueueProps
        {
            QueueName = "hl-payment-instructions-dlq",
            RetentionPeriod = Duration.Days(14),     // longer than the source queue: the clock starts at the ORIGINAL enqueue time
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true
        });

        var queue = new Queue(this, "Instructions", new QueueProps
        {
            QueueName = "hl-payment-instructions",
            VisibilityTimeout = Duration.Seconds(180),   // comfortably above the slowest observed processing time
            RetentionPeriod = Duration.Days(4),
            ReceiveMessageWaitTime = Duration.Seconds(20),   // long polling
            Encryption = QueueEncryption.SQS_MANAGED,
            EnforceSSL = true,
            DeadLetterQueue = new DeadLetterQueue { Queue = dlq, MaxReceiveCount = 5 }
        });

        // Any message in the DLQ is an incident for a payments queue.
        new Alarm(this, "DlqNotEmpty", new AlarmProps
        {
            AlarmName = "hl-payment-instructions-dlq-not-empty",
            AlarmDescription = "Messages in the payment instructions DLQ. See playbook sqs-messages-stuck-or-redelivered.",
            Metric = dlq.MetricApproximateNumberOfMessagesVisible(new MetricOptions { Period = Duration.Minutes(1) }),
            Threshold = 0,
            EvaluationPeriods = 1,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        // Delay seen by customers: oldest message older than 5 minutes for 3 periods.
        new Alarm(this, "OldestMessageAge", new AlarmProps
        {
            AlarmName = "hl-payment-instructions-age",
            Metric = queue.MetricApproximateAgeOfOldestMessage(new MetricOptions { Period = Duration.Minutes(1), Statistic = "Maximum" }),
            Threshold = 300,
            EvaluationPeriods = 3,
            ComparisonOperator = ComparisonOperator.GREATER_THAN_THRESHOLD,
            TreatMissingData = TreatMissingData.NOT_BREACHING
        });

        Amazon.CDK.Tags.Of(this).Add("Project", "cea-lab");
        Amazon.CDK.Tags.Of(this).Add("Lab", "m17-playbook-sqs");
    }
}

Complete code

A simulation that reproduces the duplicate and the fixes

This program needs no AWS account. It models an SQS queue's lending rules in virtual time (one tick is one second) with two consumers, and runs four scenarios: the duplicate-processing bug, then the fix by longer visibility timeout, the fix by an idempotency guard, and a poison message that ends in the DLQ. The model is deliberately simplified (it ignores network delay, receipt handles and approximate counts) so the cause and effect are clear.

QueueSimulation/Program.cs
var scenarios = new[]
{
    new Scenario("1 BUG: visibility 30s, work 40s, no idempotency", VisibilitySeconds: 30, WorkSeconds: 40, Idempotent: false, AlwaysFails: false, MaxReceiveCount: 5),
    new Scenario("2 FIX: visibility 60s, work 40s",                 VisibilitySeconds: 60, WorkSeconds: 40, Idempotent: false, AlwaysFails: false, MaxReceiveCount: 5),
    new Scenario("3 FIX: visibility 30s, work 40s, idempotent",     VisibilitySeconds: 30, WorkSeconds: 40, Idempotent: true,  AlwaysFails: false, MaxReceiveCount: 5),
    new Scenario("4 POISON: handler always fails, maxReceive 3",    VisibilitySeconds: 30, WorkSeconds: 5,  Idempotent: true,  AlwaysFails: true,  MaxReceiveCount: 3),
};

foreach (var s in scenarios)
{
    var r = Simulate(s);
    Console.WriteLine($"== {s.Name}");
    foreach (var line in r.Log) Console.WriteLine("   " + line);
    Console.WriteLine($"   RESULT: side effects executed = {r.SideEffects}, receives = {r.Receives}, in DLQ = {r.InDlq}");
    Console.WriteLine();
}
return 0;

static Result Simulate(Scenario s)
{
    const string id = "pay-1001";
    var visibleAt = 0;                       // virtual second at which the message is next visible
    var receiveCount = 0;
    var exists = true;                       // still in the source queue
    var inDlq = false;
    var sideEffects = 0;
    var done = new HashSet<string>();        // the idempotency store
    var log = new List<string>();
    var consumers = new[] { new Consumer("A"), new Consumer("B") };

    for (var t = 0; t <= 200; t++)
    {
        foreach (var c in consumers)
        {
            // 1. A consumer that finishes its work now.
            if (c.Busy && t >= c.FinishAt)
            {
                c.Busy = false;
                if (s.AlwaysFails)
                {
                    log.Add($"t={t,3}s {c.Name} FAILED, does not delete (message returns after its timeout)");
                }
                else if (s.Idempotent && done.Contains(id))
                {
                    log.Add($"t={t,3}s {c.Name} finished: duplicate detected, side effect SKIPPED");
                    exists = false;
                }
                else
                {
                    sideEffects++;
                    done.Add(id);
                    log.Add($"t={t,3}s {c.Name} finished: payment submitted (side effects so far: {sideEffects}), deletes message");
                    exists = false;
                }
            }

            // 2. An idle consumer polls the queue.
            if (!c.Busy && exists && t >= visibleAt)
            {
                receiveCount++;
                if (receiveCount > s.MaxReceiveCount)
                {
                    exists = false;
                    inDlq = true;
                    log.Add($"t={t,3}s {c.Name} polled: receive count would be {receiveCount} (more than {s.MaxReceiveCount}), SQS moves the message to the DLQ");
                    continue;
                }
                visibleAt = t + s.VisibilitySeconds;
                c.Busy = true;
                c.FinishAt = t + s.WorkSeconds;
                log.Add($"t={t,3}s {c.Name} received (receive count {receiveCount}), invisible until t={visibleAt}s, working {s.WorkSeconds}s");
            }
        }
    }
    return new Result(log, sideEffects, receiveCount, inDlq);
}

record Scenario(string Name, int VisibilitySeconds, int WorkSeconds, bool Idempotent, bool AlwaysFails, int MaxReceiveCount);
record Result(List<string> Log, int SideEffects, int Receives, bool InDlq);

class Consumer(string name)
{
    public string Name { get; } = name;
    public bool Busy { get; set; }
    public int FinishAt { get; set; }
}

Create a console project targeting net10.0, save the code as Program.cs and run dotnet run. In scenario 1 you should see two "payment submitted" lines (side effects = 2). Scenario 2 shows one, scenario 3 shows one plus a "duplicate detected" line, and scenario 4 ends with the message in the DLQ after three attempts.

A real consumer with a visibility heartbeat and an idempotency guard

This sketch shows the production shape with AWSSDK.SQS v4. It extends the visibility timeout while work is running and skips work already done. Replace the in-memory idempotency store with a conditional write to DynamoDB, as shown in Idempotency in .NET.

PaymentWorker.cs
using Amazon.SQS;
using Amazon.SQS.Model;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

public interface IIdempotencyStore
{
    // Returns true if this caller is the first to claim the key, false if it was already claimed.
    Task<bool> TryClaimAsync(string key, CancellationToken ct);
}

public sealed class PaymentWorker(IAmazonSQS sqs, IIdempotencyStore store, ILogger<PaymentWorker> log, string queueUrl)
    : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var response = await sqs.ReceiveMessageAsync(new ReceiveMessageRequest
            {
                QueueUrl = queueUrl,
                MaxNumberOfMessages = 10,
                WaitTimeSeconds = 20,                                   // long polling: fewer empty receives, lower cost
                MessageSystemAttributeNames = new List<string> { "ApproximateReceiveCount" }
            }, stoppingToken);

            foreach (var message in response.Messages ?? new List<Message>())
                await HandleAsync(message, stoppingToken);
        }
    }

    private async Task HandleAsync(Message message, CancellationToken ct)
    {
        message.Attributes.TryGetValue("ApproximateReceiveCount", out var receives);

        // Heartbeat: keep the message invisible while the work is genuinely running.
        using var heartbeatCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
        var heartbeat = Task.Run(async () =>
        {
            while (!heartbeatCts.IsCancellationRequested)
            {
                await Task.Delay(TimeSpan.FromSeconds(20), heartbeatCts.Token);
                await sqs.ChangeMessageVisibilityAsync(queueUrl, message.ReceiptHandle, 60, heartbeatCts.Token);
            }
        }, heartbeatCts.Token);

        try
        {
            // The SQS MessageId is stable across redeliveries of one message. If the PRODUCER may also send
            // duplicates (a retried HTTP call), key on a business identifier carried in the body instead.
            if (!await store.TryClaimAsync($"payment-instruction:{message.MessageId}", ct))
            {
                log.LogInformation("Duplicate delivery skipped, message {MessageId}, receive count {Receives}", message.MessageId, receives);
            }
            else
            {
                await SubmitToClearingAsync(message.Body, ct);          // the real side effect
            }

            await sqs.DeleteMessageAsync(queueUrl, message.ReceiptHandle, ct);   // delete ONLY after success
        }
        catch (Exception ex)
        {
            // Do not delete: the message returns after the visibility timeout and counts towards maxReceiveCount.
            log.LogError(ex, "Handler failed for message {MessageId}, receive count {Receives}", message.MessageId, receives);
        }
        finally
        {
            await heartbeatCts.CancelAsync();
            try { await heartbeat; } catch (OperationCanceledException) { }
        }
    }

    private static Task SubmitToClearingAsync(string body, CancellationToken ct) => Task.CompletedTask;   // placeholder
}

For Lambda consumers, enable Report batch item failures on the event source mapping and return the failed IDs so one bad record does not retry the whole batch:

Function.cs
using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

public class Function
{
    public async Task<SQSBatchResponse> Handle(SQSEvent evnt, ILambdaContext context)
    {
        var failures = new List<SQSBatchResponse.BatchItemFailure>();
        foreach (var record in evnt.Records)
        {
            try { await ProcessAsync(record.Body); }
            catch (Exception ex)
            {
                context.Logger.LogError($"Failed {record.MessageId}: {ex.Message}");
                failures.Add(new SQSBatchResponse.BatchItemFailure { ItemIdentifier = record.MessageId });
            }
        }
        return new SQSBatchResponse(failures);    // only these return to the queue
    }

    private static Task ProcessAsync(string body) => Task.CompletedTask;   // placeholder
}

Code explained line by line

  1. The simulation's visibleAt is the second at which the message can next be received. Receiving sets it to now plus the visibility timeout. This one variable is the entire lending mechanism.
  2. A consumer that finishes calls "delete" by setting exists = false. In scenario 1 consumer A finishes at second 40 and deletes, but consumer B had already received the message at second 30, so B also performs the side effect at second 70. That is the double payment.
  3. In scenario 3 the done set plays the idempotency store. B finds the key already claimed and skips the side effect, so the duplicate delivery is harmless.
  4. Scenario 4 never deletes. After each failure the timeout expires, the message returns, and when the receive count would exceed MaxReceiveCount, the message goes to the DLQ instead of looping forever.
  5. In the real worker, WaitTimeSeconds = 20 turns on long polling.
  6. MessageSystemAttributeNames with ApproximateReceiveCount asks SQS to include the receive count so logs can show how many times a message has been delivered.
  7. The heartbeat task calls ChangeMessageVisibilityAsync every 20 seconds, resetting the timeout to 60 seconds from now. This keeps a legitimately slow message invisible without a huge static timeout.
  8. TryClaimAsync is the idempotency guard. In production it is a DynamoDB conditional put, so only one caller can claim the key.
  9. DeleteMessageAsync runs only after success. If anything throws, the code does not delete, so SQS redelivers it, which is the retry mechanism.
  10. In the Lambda handler, returning SQSBatchResponse with only the failed message IDs tells the service to delete the successes and retry just the failures.

Security considerations

⚠ Learning shortcut — not for production

Inspecting messages by polling the DLQ in the console or CLI shows message bodies, which can contain personal and payment data. In production, restrict who can receive from the DLQ, do not paste bodies into tickets or chat, and log identifiers rather than payloads. The simulation also stores nothing sensitive; a real idempotency store must be durable and access controlled.

  • Least privilege on the queue. Give consumers receive, delete and change-visibility on their queue only, producers send-only, and responders a separate read-only role for the DLQ. Use a queue policy to restrict which principals and source ARNs may send.
  • Encrypt at rest and in transit. Use SQS-managed or KMS encryption and enforce TLS. With a customer managed key, consumers need kms:Decrypt and producers kms:GenerateDataKey, which is a common cause of mysterious receive failures (see KMS envelope encryption).
  • Never purge-queue during an incident, and protect the permission with an explicit deny for everyone but a break-glass role.
  • Redrive is a change. It reintroduces old instructions into the live system. For payments, confirm with the business owner that instructions are still valid (not stale, not already compensated) before redriving.
  • Poison messages can be attacks. Validate and bound message size and shape, and never execute content from a message.
  • Audit. Record message IDs moved, who redrove, when and with what outcome.

Banking use case

Harbourline's payment-instruction queue has a visibility timeout of 180 seconds derived from the 99th percentile of the partner call (measured, not guessed), a heartbeat for exceptional cases, maxReceiveCount of 5, a 14-day DLQ and an alarm that pages on any DLQ message. Each instruction carries a business identifier, and the worker claims it with a conditional write before calling the clearing partner, so a duplicate delivery cannot cause a second payment. During the 15:20 incident, the team compared received and deleted counts, found receive counts of 2 on slow messages, raised the visibility timeout, added the idempotency guard, and redrove the three poison instructions from the DLQ at one message per second after correcting a malformed currency field. The audit record lists the three message IDs, the correction approved by the payments owner, and the redrive task handle.

Common mistakes

  • Visibility timeout left at the 30-second default while the work takes longer.
  • No idempotency, assuming SQS delivers once.
  • No DLQ, or a DLQ nobody watches.
  • DLQ retention equal to or shorter than the source queue, so messages expire right after moving.
  • Deleting before the work finishes, which loses messages on crash.
  • Retrying a whole Lambda batch because of one bad record.
  • maxReceiveCount of 1 with Lambda, so a throttle sends good messages to the DLQ.
  • Using one FIFO message group for everything, so one poison message blocks all traffic.
  • Purging a queue to "fix" it, deleting evidence and good messages.

Troubleshooting

SymptomLikely causeFix
Visible messages growing, consumers idleConsumers cannot receive (permission, KMS, wrong URL, VPC endpoint missing)Check consumer logs for AccessDenied or timeouts, the queue policy and the endpoint
NotVisible stays high, never dropsConsumers receive but never deleteFix the code path, delete on success
Same message repeats with rising receive countHandler failing or processing slower than the visibility timeoutCheck logs by message ID, raise the timeout or add a heartbeat
Many messages in the DLQ after a short outageDownstream was down and maxReceiveCount was lowFix the dependency, redrive at a low rate, raise maxReceiveCount
Lambda errors "throttled" and messages in the DLQConcurrency limit reachedAdjust reserved and maximum concurrency, raise maxReceiveCount
FIFO queue only delivers part of the messagesA failing message blocks its groupFix, delete or redrive the blocking message, split group IDs
Redrive fails or does nothingDLQ and source types differ, or missing permissionMatch standard to standard, FIFO to FIFO, check redrive permissions
DLQ message expired quicklyRetention counted from original enqueue timeSet DLQ retention longer than the source queue
Duplicate effects despite a DLQConsumer not idempotentAdd a durable idempotency guard

Cost and cleanup

Cost label: free. The simulation needs no AWS resources. A sandbox queue and a handful of messages cost far less than a cent (⚠ VERIFY current pricing, which has a monthly free tier), and the alarms in this lesson can incur a small charge, so delete them afterwards.

# Delete the sandbox queues and alarms
aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions --profile learn-sandbox
aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/hl-payment-instructions-dlq --profile learn-sandbox
aws cloudwatch delete-alarms --alarm-names hl-payment-instructions-dlq-not-empty hl-payment-instructions-age --profile learn-sandbox

# Confirm nothing tagged for this lesson remains
aws resourcegroupstaggingapi get-resources --tag-filters Key=Lab,Values=m17-playbook-sqs --profile learn-sandbox

Exercises

  • 🟢 Guided: Run the simulation. For scenario 1, write the exact second each consumer received the message and each side effect happened. Then change VisibilitySeconds to 45 and 41 and explain the result.
  • 🟡 Challenge: Add a scenario with a heartbeat: the consumer extends visibility by 30 seconds every 20 seconds while working. Show that with a 30-second base timeout and 90 seconds of work there is still only one receive.
  • 🔴 Live problem: "A FIFO queue of customer notifications stopped delivering for one customer's messages two hours ago; others are fine. DLQ is empty. The consumer logs show the same message failing every 30 seconds." Explain what is happening, what you check, and what you do about it.

Assignment

Write an SQS runbook page for your portfolio: the message lifecycle diagram, the metric table, the symptom-to-cause table, the safe DLQ inspection and redrive commands, the alarm definitions and the idempotency approach for one capstone queue. Commit as runbooks/sqs-stuck-or-redelivered.md with fictional identifiers.

Solutions

Knowledge check

Interview questions

  • Messages in an SQS queue are processed twice. Why, and what do you do? At-least-once delivery, plus a visibility timeout shorter than processing or consumer retries. Make handlers idempotent, align the timeout and use a heartbeat if needed.
  • How do you handle poison messages? A redrive policy with a sensible maxReceiveCount, a DLQ with an alarm, inspection without deleting, fixing the cause and controlled redrive.
  • How do you set the visibility timeout? From measured processing time at a high percentile with headroom, six times the function timeout for Lambda (⚠ VERIFY), with a heartbeat for exceptional cases.
  • What is different about FIFO queues when things fail? Ordering within a group means one failing message blocks its group, and deduplication is limited to a five-minute window.
  • How would you redrive 10,000 messages safely? Fix the cause first, ensure idempotency, redrive at a controlled rate, watch the consumer's error rate and be ready to cancel the task.

Certification practice

Completion checklist

  • I can describe the receive, invisible, delete and redeliver cycle
  • I can tell a slow consumer from a poison message using receive count
  • I ran the simulation and explained scenario 1 second by second
  • I know how to inspect a DLQ without deleting and how to redrive at a controlled rate
  • I can explain why handlers must be idempotent
  • I ran the cleanup steps and confirmed nothing tagged m17-playbook-sqs remains

Key takeaways

  • SQS lends messages: visibility timeout, receive count and maxReceiveCount explain nearly every symptom.
  • At-least-once delivery means idempotent handlers are mandatory.
  • Visibility timeout should exceed worst-case processing time, with a heartbeat for exceptions.
  • Watch the DLQ and the age of the oldest message, and inspect before redriving.
  • Never purge during an incident.

Further reading

Independent learning resource — not affiliated with, endorsed or sponsored by Amazon Web Services. AWS and related marks are trademarks of Amazon.com, Inc. or its affiliates. All banking data in examples is fictional. Sample architectures are for learning and are not automatically compliant with GDPR, PCI DSS or any banking regulation.

On this page